diff --git a/docs/dag/00-INDEX.md b/docs/dag/00-INDEX.md new file mode 100644 index 000000000..97c6363e6 --- /dev/null +++ b/docs/dag/00-INDEX.md @@ -0,0 +1,83 @@ +# Neural Self-Learning DAG Implementation Plan + +## Project Overview + +This document set provides a complete implementation plan for integrating a Neural Self-Learning DAG system into RuVector-Postgres, with optional QuDAG distributed consensus integration. + +## Document Index + +| Document | Description | Priority | +|----------|-------------|----------| +| [01-ARCHITECTURE.md](./01-ARCHITECTURE.md) | System architecture and component overview | P0 | +| [02-DAG-ATTENTION-MECHANISMS.md](./02-DAG-ATTENTION-MECHANISMS.md) | 7 specialized DAG attention implementations | P0 | +| [03-SONA-INTEGRATION.md](./03-SONA-INTEGRATION.md) | Self-Optimizing Neural Architecture integration | P0 | +| [04-POSTGRES-INTEGRATION.md](./04-POSTGRES-INTEGRATION.md) | PostgreSQL extension integration details | P0 | +| [05-QUERY-PLAN-DAG.md](./05-QUERY-PLAN-DAG.md) | Query plan as learnable DAG structure | P1 | +| [06-MINCUT-OPTIMIZATION.md](./06-MINCUT-OPTIMIZATION.md) | Min-cut based bottleneck detection | P1 | +| [07-SELF-HEALING.md](./07-SELF-HEALING.md) | Self-healing and adaptive repair | P1 | +| [08-QUDAG-INTEGRATION.md](./08-QUDAG-INTEGRATION.md) | QuDAG distributed consensus integration | P2 | +| [09-SQL-API.md](./09-SQL-API.md) | Complete SQL API specification | P0 | +| [10-TESTING-STRATEGY.md](./10-TESTING-STRATEGY.md) | Testing approach and benchmarks | P1 | +| [11-AGENT-TASKS.md](./11-AGENT-TASKS.md) | 15-agent swarm task breakdown | P0 | +| [12-MILESTONES.md](./12-MILESTONES.md) | Implementation milestones and timeline | P0 | + +## Quick Start for Agents + +1. Read [01-ARCHITECTURE.md](./01-ARCHITECTURE.md) for system overview +2. Check [11-AGENT-TASKS.md](./11-AGENT-TASKS.md) for your assigned tasks +3. Follow task-specific documents as referenced +4. Coordinate via shared memory patterns in [03-SONA-INTEGRATION.md](./03-SONA-INTEGRATION.md) + +## Project Goals + +### Primary Goals +- Create self-learning query optimization for RuVector-Postgres +- Implement 7 DAG-centric attention mechanisms +- Integrate SONA two-tier learning system +- Provide adaptive cost estimation +- Enable bottleneck detection via min-cut analysis + +### Secondary Goals +- QuDAG distributed consensus for federated learning +- Self-healing index maintenance +- HDC state compression for efficient sync +- Production-ready SQL API + +## Success Metrics + +| Metric | Target | Measurement | +|--------|--------|-------------| +| Query latency improvement | 30-50% | Benchmark suite | +| Pattern recall accuracy | >95% | Test coverage | +| Learning overhead | <5% | Per-query timing | +| Bottleneck detection | O(n^0.12) | Algorithmic analysis | +| Memory overhead | <100MB | Per-table measurement | + +## Dependencies + +### Required Crates (Internal) +- `ruvector-postgres` - PostgreSQL extension framework +- `ruvector-attention` - 39 attention mechanisms +- `ruvector-gnn` - Graph neural network layers +- `ruvector-graph` - Query execution DAG +- `ruvector-mincut` - Subpolynomial min-cut +- `ruvector-nervous-system` - BTSP, HDC, spiking networks +- `sona` - Self-Optimizing Neural Architecture + +### Required Crates (External) +- `pgrx` - PostgreSQL Rust extension framework +- `dashmap` - Concurrent hashmap +- `parking_lot` - Fast synchronization primitives +- `ndarray` - N-dimensional arrays +- `rayon` - Parallel iterators + +### Optional (QuDAG Integration) +- `qudag` - Quantum-resistant DAG consensus +- `ml-kem` - Post-quantum key encapsulation +- `ml-dsa` - Post-quantum signatures + +## Version + +- Plan Version: 1.0.0 +- Target RuVector Version: 0.5.0 +- Last Updated: 2025-12-29 diff --git a/docs/dag/01-ARCHITECTURE.md b/docs/dag/01-ARCHITECTURE.md new file mode 100644 index 000000000..0fb2dc9ec --- /dev/null +++ b/docs/dag/01-ARCHITECTURE.md @@ -0,0 +1,484 @@ +# Neural Self-Learning DAG Architecture + +## Overview + +The Neural Self-Learning DAG system transforms RuVector-Postgres from a static query executor into an adaptive system that learns optimal configurations from query patterns. + +## System Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ NEURAL DAG RUVECTOR-POSTGRES │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ SQL INTERFACE LAYER │ │ +│ │ ruvector_enable_neural_dag() | ruvector_dag_patterns() | ... │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ QUERY OPTIMIZER LAYER │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │ +│ │ │ Pattern │ │ Attention │ │ Cost │ │ Plan │ │ │ +│ │ │ Matcher │ │ Selector │ │ Estimator │ │ Rewriter │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ DAG ATTENTION LAYER │ │ +│ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ │ +│ │ │Topological│ │ Causal │ │ Critical │ │ MinCut │ │ │ +│ │ │ Attention │ │ Cone │ │ Path │ │ Gated │ │ │ +│ │ └───────────┘ └───────────┘ └───────────┘ └───────────┘ │ │ +│ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ │ +│ │ │Hierarchic │ │ Parallel │ │ Temporal │ │ │ +│ │ │ Lorentz │ │ Branch │ │ BTSP │ │ │ +│ │ └───────────┘ └───────────┘ └───────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ SONA LEARNING LAYER │ │ +│ │ ┌─────────────────────────────────────────────────────────────┐ │ │ +│ │ │ INSTANT LOOP (<100μs) BACKGROUND LOOP (hourly) │ │ │ +│ │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ +│ │ │ │ MicroLoRA │ │ BaseLoRA │ │ │ │ +│ │ │ │ (rank 1-2) │ │ (rank 8) │ │ │ │ +│ │ │ └─────────────┘ └─────────────┘ │ │ │ +│ │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ +│ │ │ │ Trajectory │ ──────────────► │ ReasoningBk │ │ │ │ +│ │ │ │ Buffer │ │ (K-means) │ │ │ │ +│ │ │ └─────────────┘ └─────────────┘ │ │ │ +│ │ │ ┌─────────────┐ │ │ │ +│ │ │ │ EWC++ │ │ │ │ +│ │ │ │ (forgetting)│ │ │ │ +│ │ │ └─────────────┘ │ │ │ +│ │ └─────────────────────────────────────────────────────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ OPTIMIZATION LAYER │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │ +│ │ │ MinCut │ │ HDC │ │ BTSP │ │ Self- │ │ │ +│ │ │ Analysis │ │ State │ │ Memory │ │ Healing │ │ │ +│ │ │ O(n^0.12) │ │ Compression │ │ One-Shot │ │ Engine │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ STORAGE LAYER │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │ +│ │ │ Pattern │ │ Embedding │ │ Trajectory │ │ Index │ │ │ +│ │ │ Store │ │ Cache │ │ History │ │ Metadata │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ OPTIONAL: QUDAG CONSENSUS LAYER │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │ +│ │ │ Federated │ │ Pattern │ │ ML-DSA │ │ rUv │ │ │ +│ │ │ Learning │ │ Consensus │ │ Signatures │ │ Tokens │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## Component Descriptions + +### 1. SQL Interface Layer + +Provides PostgreSQL-native functions for interacting with the Neural DAG system. + +**Key Components:** +- `ruvector_enable_neural_dag()` - Enable learning for a table +- `ruvector_dag_patterns()` - View learned patterns +- `ruvector_attention_*()` - DAG attention functions +- `ruvector_dag_learn()` - Trigger learning cycle + +**Location:** `crates/ruvector-postgres/src/dag/operators.rs` + +### 2. Query Optimizer Layer + +Intercepts queries and applies learned optimizations. + +**Key Components:** +- **Pattern Matcher**: Finds similar past query patterns via cosine similarity +- **Attention Selector**: UCB bandit for choosing optimal attention type +- **Cost Estimator**: Adaptive cost model with micro-LoRA updates +- **Plan Rewriter**: Applies learned operator ordering and parameters + +**Location:** `crates/ruvector-postgres/src/dag/optimizer.rs` + +### 3. DAG Attention Layer + +Seven specialized attention mechanisms for DAG structures. + +| Attention Type | Use Case | Complexity | +|----------------|----------|------------| +| Topological | Respect DAG ordering | O(n·k) | +| Causal Cone | Distance-weighted ancestors | O(n·d) | +| Critical Path | Focus on bottlenecks | O(n + critical_len) | +| MinCut Gated | Gate by criticality | O(n^0.12 + n·k) | +| Hierarchical Lorentz | Deep nesting | O(n·d) | +| Parallel Branch | Coordinate branches | O(n·b) | +| Temporal BTSP | Time-correlated patterns | O(n·w) | + +**Location:** `crates/ruvector-postgres/src/dag/attention/` + +### 4. SONA Learning Layer + +Two-tier learning system for continuous optimization. + +**Instant Loop (per-query):** +- MicroLoRA adaptation (rank 1-2) +- Trajectory recording +- <100μs overhead + +**Background Loop (hourly):** +- K-means++ pattern extraction +- BaseLoRA updates (rank 8) +- EWC++ constraint application + +**Location:** `crates/ruvector-postgres/src/dag/learning/` + +### 5. Optimization Layer + +Advanced optimization components. + +**Key Components:** +- **MinCut Analysis**: O(n^0.12) bottleneck detection +- **HDC State**: 10K-bit hypervector compression +- **BTSP Memory**: One-shot pattern recall +- **Self-Healing**: Proactive index repair + +**Location:** `crates/ruvector-postgres/src/dag/optimization/` + +### 6. Storage Layer + +Persistent storage for learned patterns and state. + +**Key Components:** +- **Pattern Store**: DashMap + PostgreSQL tables +- **Embedding Cache**: LRU cache for hot embeddings +- **Trajectory History**: Ring buffer for recent queries +- **Index Metadata**: Pattern-to-index mappings + +**Location:** `crates/ruvector-postgres/src/dag/storage/` + +### 7. QuDAG Consensus Layer (Optional) + +Distributed learning via quantum-resistant consensus. + +**Key Components:** +- **Federated Learning**: Privacy-preserving pattern sharing +- **Pattern Consensus**: QR-Avalanche for pattern validation +- **ML-DSA Signatures**: Quantum-resistant pattern signing +- **rUv Tokens**: Incentivize learning contributions + +**Location:** `crates/ruvector-postgres/src/dag/qudag/` + +## Data Flow + +### Query Execution Flow + +``` +SQL Query + │ + ▼ +┌─────────────────────────────────────┐ +│ 1. Pattern Matching │ +│ - Embed query plan │ +│ - Find similar patterns in │ +│ ReasoningBank (cosine sim) │ +│ - Return top-k matches │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 2. Optimization Decision │ +│ - If pattern found (conf > 0.8): │ +│ Apply learned configuration │ +│ - Else: │ +│ Use defaults + micro-LoRA │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 3. Attention Selection │ +│ - UCB bandit selects attention │ +│ - Based on query pattern type │ +│ - Exploration vs exploitation │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 4. Plan Execution │ +│ - Execute with optimized params │ +│ - Record operator timings │ +│ - Track intermediate results │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 5. Trajectory Recording │ +│ - Store query embedding │ +│ - Store operator activations │ +│ - Store outcome metrics │ +│ - Compute quality score │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 6. Instant Learning │ +│ - MicroLoRA gradient accumulate │ +│ - Auto-flush at 100 queries │ +│ - Update attention selector │ +└─────────────────────────────────────┘ +``` + +### Learning Cycle Flow + +``` +Hourly Trigger + │ + ▼ +┌─────────────────────────────────────┐ +│ 1. Drain Trajectory Buffer │ +│ - Collect 1000+ trajectories │ +│ - Filter by quality threshold │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 2. K-means++ Clustering │ +│ - 100 clusters │ +│ - Deterministic initialization │ +│ - Max 100 iterations │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 3. Pattern Extraction │ +│ - Compute cluster centroids │ +│ - Extract optimal parameters │ +│ - Calculate confidence scores │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 4. EWC++ Constraint Check │ +│ - Compute Fisher information │ +│ - Apply forgetting prevention │ +│ - Detect task boundaries │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 5. BaseLoRA Update │ +│ - Apply constrained gradients │ +│ - Update all layers │ +│ - Merge weights if needed │ +└─────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────┐ +│ 6. ReasoningBank Update │ +│ - Store new patterns │ +│ - Consolidate similar patterns │ +│ - Evict low-confidence patterns │ +└─────────────────────────────────────┘ +``` + +## Module Dependencies + +``` +ruvector-postgres/src/dag/ +├── mod.rs # Module root, re-exports +├── operators.rs # SQL function definitions +│ +├── attention/ +│ ├── mod.rs # Attention trait and registry +│ ├── topological.rs # TopologicalAttention +│ ├── causal_cone.rs # CausalConeAttention +│ ├── critical_path.rs # CriticalPathAttention +│ ├── mincut_gated.rs # MinCutGatedAttention +│ ├── hierarchical.rs # HierarchicalLorentzAttention +│ ├── parallel_branch.rs # ParallelBranchAttention +│ ├── temporal_btsp.rs # TemporalBTSPAttention +│ └── ensemble.rs # EnsembleAttention +│ +├── learning/ +│ ├── mod.rs # Learning coordinator +│ ├── sona_engine.rs # SONA integration wrapper +│ ├── trajectory.rs # Trajectory buffer +│ ├── patterns.rs # Pattern extraction +│ ├── reasoning_bank.rs # Pattern storage +│ ├── ewc.rs # EWC++ integration +│ └── attention_selector.rs # UCB bandit selector +│ +├── optimizer/ +│ ├── mod.rs # Optimizer coordinator +│ ├── pattern_matcher.rs # Pattern matching +│ ├── cost_estimator.rs # Adaptive costs +│ └── plan_rewriter.rs # Plan transformation +│ +├── optimization/ +│ ├── mod.rs # Optimization utilities +│ ├── mincut.rs # Min-cut integration +│ ├── hdc_state.rs # HDC compression +│ ├── btsp_memory.rs # BTSP one-shot +│ └── self_healing.rs # Self-healing engine +│ +├── storage/ +│ ├── mod.rs # Storage coordinator +│ ├── pattern_store.rs # Pattern persistence +│ ├── embedding_cache.rs # Embedding LRU +│ └── trajectory_store.rs # Trajectory history +│ +├── qudag/ +│ ├── mod.rs # QuDAG integration +│ ├── federated.rs # Federated learning +│ ├── consensus.rs # Pattern consensus +│ ├── signatures.rs # ML-DSA signing +│ └── tokens.rs # rUv token interface +│ +└── types/ + ├── mod.rs # Type definitions + ├── neural_plan.rs # NeuralDagPlan + ├── trajectory.rs # DagTrajectory + ├── pattern.rs # LearnedDagPattern + └── metrics.rs # ExecutionMetrics +``` + +## Configuration + +### Default Configuration + +```rust +pub struct NeuralDagConfig { + // Learning + pub learning_enabled: bool, // true + pub max_trajectories: usize, // 10000 + pub pattern_clusters: usize, // 100 + pub quality_threshold: f32, // 0.3 + pub background_interval_ms: u64, // 3600000 (1 hour) + + // Attention + pub default_attention: DagAttentionType, // Topological + pub attention_exploration: f32, // 0.1 + pub ucb_exploration_c: f32, // 1.414 + + // SONA + pub micro_lora_rank: usize, // 2 + pub micro_lora_lr: f32, // 0.002 + pub base_lora_rank: usize, // 8 + pub base_lora_lr: f32, // 0.001 + + // EWC++ + pub ewc_lambda: f32, // 2000.0 + pub ewc_max_lambda: f32, // 15000.0 + pub ewc_fisher_decay: f32, // 0.999 + + // MinCut + pub mincut_enabled: bool, // true + pub mincut_threshold: f32, // 0.5 + + // HDC + pub hdc_dimensions: usize, // 10000 + + // Self-Healing + pub healing_enabled: bool, // true + pub healing_check_interval_ms: u64, // 300000 (5 min) +} +``` + +### PostgreSQL GUC Variables + +```sql +-- Enable/disable neural DAG +SET ruvector.neural_dag_enabled = true; + +-- Learning parameters +SET ruvector.dag_learning_rate = 0.002; +SET ruvector.dag_pattern_clusters = 100; +SET ruvector.dag_quality_threshold = 0.3; + +-- Attention parameters +SET ruvector.dag_attention_type = 'auto'; +SET ruvector.dag_attention_exploration = 0.1; + +-- EWC parameters +SET ruvector.dag_ewc_lambda = 2000.0; + +-- MinCut parameters +SET ruvector.dag_mincut_enabled = true; +SET ruvector.dag_mincut_threshold = 0.5; +``` + +## Performance Targets + +| Operation | Target Latency | Notes | +|-----------|----------------|-------| +| Pattern matching | <1ms | Top-5 similar patterns | +| Attention computation | <500μs | Per operator | +| MicroLoRA forward | <100μs | Per query | +| Trajectory recording | <50μs | Non-blocking | +| Background learning | <5s | 1000 trajectories | +| MinCut analysis | <10ms | O(n^0.12) | +| HDC encoding | <100μs | 10K dimensions | + +## Memory Budget + +| Component | Budget | Notes | +|-----------|--------|-------| +| Pattern Store | 50MB | ~1000 patterns per table | +| Embedding Cache | 20MB | LRU for hot embeddings | +| Trajectory Buffer | 20MB | 10K trajectories | +| MicroLoRA | 10KB | Per active query | +| BaseLoRA | 400KB | Per table | +| HDC State | 1.2KB | Per state snapshot | + +**Total per table:** ~100MB maximum + +## Thread Safety + +All components use thread-safe primitives: + +- `DashMap` for concurrent pattern storage +- `parking_lot::RwLock` for embedding cache +- `crossbeam::ArrayQueue` for trajectory buffer +- `AtomicU64` for counters and statistics +- PostgreSQL background workers for learning cycles + +## Error Handling + +```rust +pub enum NeuralDagError { + // Configuration errors + InvalidConfig(String), + TableNotEnabled(String), + + // Learning errors + InsufficientTrajectories, + PatternExtractionFailed, + EwcConstraintViolation, + + // Attention errors + AttentionComputationFailed, + InvalidDagStructure, + + // Storage errors + PatternStoreFull, + EmbeddingCacheMiss, + + // MinCut errors + MinCutComputationFailed, + GraphDisconnected, + + // QuDAG errors (optional) + ConsensusTimeout, + SignatureVerificationFailed, +} +``` + +All errors are logged and non-fatal - the system falls back to default behavior on error. diff --git a/docs/dag/02-DAG-ATTENTION-MECHANISMS.md b/docs/dag/02-DAG-ATTENTION-MECHANISMS.md new file mode 100644 index 000000000..f46b08126 --- /dev/null +++ b/docs/dag/02-DAG-ATTENTION-MECHANISMS.md @@ -0,0 +1,1236 @@ +# DAG Attention Mechanisms + +## Overview + +This document specifies seven specialized attention mechanisms designed for Directed Acyclic Graph (DAG) structures. Each mechanism leverages unique DAG properties for specific optimization scenarios. + +## Attention Trait Definition + +```rust +/// Core trait for all DAG attention mechanisms +pub trait DagAttention: Send + Sync { + /// Compute attention weights for a query node over its context + fn forward( + &self, + query: &DagNode, + context: &DagContext, + config: &AttentionConfig, + ) -> Result; + + /// Update internal state based on execution feedback + fn update(&mut self, feedback: &AttentionFeedback) -> Result<(), AttentionError>; + + /// Get attention type identifier + fn attention_type(&self) -> DagAttentionType; + + /// Estimated computation complexity + fn complexity(&self, context_size: usize) -> usize; +} + +/// Output from attention computation +pub struct AttentionOutput { + /// Attention weights (sum to 1.0) + pub weights: Vec, + + /// Weighted aggregation of context values + pub aggregated: Vec, + + /// Auxiliary information for learning + pub metadata: AttentionMetadata, +} + +/// Context for DAG attention +pub struct DagContext { + /// All nodes in the DAG + pub nodes: Vec, + + /// Adjacency list (node_id -> children) + pub edges: HashMap>, + + /// Reverse adjacency (node_id -> parents) + pub reverse_edges: HashMap>, + + /// Node depths (topological distance from roots) + pub depths: HashMap, + + /// Optional: timestamps for temporal attention + pub timestamps: Option>, + + /// Optional: min-cut criticality scores + pub criticalities: Option>, +} +``` + +--- + +## 1. Topological Attention + +### Purpose +Respects DAG ordering by allowing nodes to only attend to their ancestors. This maintains causal consistency in query plans. + +### Algorithm + +```rust +pub struct TopologicalAttention { + /// Hidden dimension for projections + hidden_dim: usize, + + /// Number of attention heads + num_heads: usize, + + /// Query/Key/Value projection weights + w_query: Array2, // [hidden_dim, hidden_dim] + w_key: Array2, + w_value: Array2, + + /// Precomputed ancestor masks (lazily computed) + ancestor_cache: DashMap, +} + +impl TopologicalAttention { + pub fn forward(&self, query_node: &DagNode, ctx: &DagContext) -> AttentionOutput { + // 1. Get or compute ancestor mask + let ancestors = self.get_ancestors(query_node.id, ctx); + + // 2. Project query + let q = self.project_query(&query_node.embedding); + + // 3. Compute attention scores (only for ancestors) + let mut scores = Vec::with_capacity(ctx.nodes.len()); + let scale = (self.hidden_dim as f32).sqrt(); + + for (i, node) in ctx.nodes.iter().enumerate() { + if ancestors.get(i).unwrap_or(false) { + let k = self.project_key(&node.embedding); + let score = dot(&q, &k) / scale; + scores.push(score); + } else { + scores.push(f32::NEG_INFINITY); // Mask non-ancestors + } + } + + // 4. Softmax + let weights = softmax(&scores); + + // 5. Weighted aggregation of values + let mut aggregated = vec![0.0; self.hidden_dim]; + for (i, node) in ctx.nodes.iter().enumerate() { + if weights[i] > 1e-8 { + let v = self.project_value(&node.embedding); + for (j, val) in v.iter().enumerate() { + aggregated[j] += weights[i] * val; + } + } + } + + AttentionOutput { + weights, + aggregated, + metadata: AttentionMetadata::new(DagAttentionType::Topological), + } + } + + fn get_ancestors(&self, node_id: NodeId, ctx: &DagContext) -> BitVec { + if let Some(cached) = self.ancestor_cache.get(&node_id) { + return cached.clone(); + } + + // BFS to find all ancestors + let mut ancestors = BitVec::repeat(false, ctx.nodes.len()); + let mut queue = VecDeque::new(); + + // Start with direct parents + if let Some(parents) = ctx.reverse_edges.get(&node_id) { + for &parent in parents { + queue.push_back(parent); + } + } + + while let Some(current) = queue.pop_front() { + let idx = ctx.node_index(current); + if !ancestors.get(idx).unwrap_or(false) { + ancestors.set(idx, true); + if let Some(parents) = ctx.reverse_edges.get(¤t) { + for &parent in parents { + queue.push_back(parent); + } + } + } + } + + self.ancestor_cache.insert(node_id, ancestors.clone()); + ancestors + } +} +``` + +### Complexity +- Time: O(n·k) where n = nodes, k = average ancestors +- Space: O(n) for ancestor mask + +### SQL Interface + +```sql +-- Compute topological attention +SELECT ruvector_attention_topological( + query_embedding, -- VECTOR: query node embedding + ancestor_embeddings, -- VECTOR[]: ancestor embeddings + '{"num_heads": 8, "hidden_dim": 256}'::jsonb -- config +) AS attention_weights; +``` + +--- + +## 2. Causal Cone Attention + +### Purpose +Extends topological attention with distance-weighted decay. Closer ancestors get more attention. + +### Algorithm + +```rust +pub struct CausalConeAttention { + hidden_dim: usize, + num_heads: usize, + + /// Decay rate per DAG hop + decay_rate: f32, + + /// Maximum lookback depth + max_depth: usize, + + /// Projection weights + w_query: Array2, + w_key: Array2, + w_value: Array2, +} + +impl CausalConeAttention { + pub fn forward(&self, query_node: &DagNode, ctx: &DagContext) -> AttentionOutput { + let query_depth = ctx.depths[&query_node.id]; + + // 1. Compute distances from query to all ancestors + let distances = self.compute_distances(query_node.id, ctx); + + // 2. Project query + let q = self.project_query(&query_node.embedding); + let scale = (self.hidden_dim as f32).sqrt(); + + // 3. Compute distance-weighted attention + let mut scores = Vec::with_capacity(ctx.nodes.len()); + + for (i, node) in ctx.nodes.iter().enumerate() { + if let Some(&dist) = distances.get(&node.id) { + if dist > 0 && dist <= self.max_depth { + let k = self.project_key(&node.embedding); + let base_score = dot(&q, &k) / scale; + + // Exponential decay with distance + let decay = (-self.decay_rate * dist as f32).exp(); + scores.push(base_score * decay); + } else { + scores.push(f32::NEG_INFINITY); + } + } else { + scores.push(f32::NEG_INFINITY); + } + } + + // 4. Softmax and aggregate + let weights = softmax(&scores); + let aggregated = self.aggregate_values(&weights, ctx); + + AttentionOutput { + weights, + aggregated, + metadata: AttentionMetadata::new(DagAttentionType::CausalCone), + } + } + + fn compute_distances(&self, from: NodeId, ctx: &DagContext) -> HashMap { + let mut distances = HashMap::new(); + let mut queue = VecDeque::new(); + + // BFS from query node going backward + if let Some(parents) = ctx.reverse_edges.get(&from) { + for &parent in parents { + queue.push_back((parent, 1)); + } + } + + while let Some((current, dist)) = queue.pop_front() { + if dist > self.max_depth { + continue; + } + + if !distances.contains_key(¤t) { + distances.insert(current, dist); + + if let Some(parents) = ctx.reverse_edges.get(¤t) { + for &parent in parents { + queue.push_back((parent, dist + 1)); + } + } + } + } + + distances + } +} +``` + +### Configuration + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `decay_rate` | 0.1 | Exponential decay per hop | +| `max_depth` | 10 | Maximum ancestor distance | + +### SQL Interface + +```sql +SELECT ruvector_attention_causal_cone( + query_embedding, + ancestor_embeddings, + ancestor_distances, -- INT[]: hop distances + 0.1, -- decay_rate + 10 -- max_depth +) AS attention_weights; +``` + +--- + +## 3. Critical Path Attention + +### Purpose +Focuses attention on DAG critical path (longest path). Useful for identifying and optimizing bottleneck operators. + +### Algorithm + +```rust +pub struct CriticalPathAttention { + hidden_dim: usize, + num_heads: usize, + + /// Boost factor for critical path nodes + critical_path_boost: f32, + + /// Projection weights + w_query: Array2, + w_key: Array2, + w_value: Array2, + + /// Cached critical paths + critical_path_cache: DashMap>, +} + +impl CriticalPathAttention { + pub fn forward(&self, query_node: &DagNode, ctx: &DagContext) -> AttentionOutput { + // 1. Compute critical path through query node + let critical_path = self.find_critical_path(query_node.id, ctx); + + // 2. Project query + let q = self.project_query(&query_node.embedding); + let scale = (self.hidden_dim as f32).sqrt(); + + // 3. Compute attention with critical path boost + let ancestors = self.get_ancestors(query_node.id, ctx); + let mut scores = Vec::with_capacity(ctx.nodes.len()); + + for (i, node) in ctx.nodes.iter().enumerate() { + if ancestors.contains(&node.id) { + let k = self.project_key(&node.embedding); + let mut score = dot(&q, &k) / scale; + + // Boost critical path nodes + if critical_path.contains(&node.id) { + score += self.critical_path_boost; + } + + scores.push(score); + } else { + scores.push(f32::NEG_INFINITY); + } + } + + let weights = softmax(&scores); + let aggregated = self.aggregate_values(&weights, ctx); + + AttentionOutput { + weights, + aggregated, + metadata: AttentionMetadata::with_critical_path(critical_path), + } + } + + fn find_critical_path(&self, through: NodeId, ctx: &DagContext) -> HashSet { + if let Some(cached) = self.critical_path_cache.get(&through) { + return cached.clone(); + } + + // Find longest path through this node using DP + let mut longest_to = HashMap::new(); // longest path ending at node + let mut longest_from = HashMap::new(); // longest path starting from node + + // Topological order + let topo_order = self.topological_sort(ctx); + + // Forward pass: longest path TO each node + for &node in &topo_order { + let mut max_len = 0; + if let Some(parents) = ctx.reverse_edges.get(&node) { + for &parent in parents { + max_len = max_len.max(longest_to.get(&parent).unwrap_or(&0) + 1); + } + } + longest_to.insert(node, max_len); + } + + // Backward pass: longest path FROM each node + for &node in topo_order.iter().rev() { + let mut max_len = 0; + if let Some(children) = ctx.edges.get(&node) { + for &child in children { + max_len = max_len.max(longest_from.get(&child).unwrap_or(&0) + 1); + } + } + longest_from.insert(node, max_len); + } + + // Find nodes on critical path through 'through' + let total_through = longest_to[&through] + longest_from[&through]; + let global_longest = topo_order.iter() + .map(|n| longest_to[n] + longest_from[n]) + .max() + .unwrap_or(0); + + let mut critical = HashSet::new(); + if total_through == global_longest { + // 'through' is on a global critical path + for &node in &topo_order { + if longest_to[&node] + longest_from[&node] == global_longest { + critical.insert(node); + } + } + } + + self.critical_path_cache.insert(through, critical.clone()); + critical + } +} +``` + +### Configuration + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `critical_path_boost` | 1.5 | Additive boost for critical nodes | + +### SQL Interface + +```sql +SELECT ruvector_attention_critical_path( + query_embedding, + ancestor_embeddings, + is_critical, -- BOOLEAN[]: critical path membership + 1.5 -- boost factor +) AS attention_weights; +``` + +--- + +## 4. MinCut Gated Attention + +### Purpose +Uses min-cut analysis to gate information flow. Focuses learning on bottleneck operators that dominate execution time. + +### Algorithm + +```rust +pub struct MinCutGatedAttention { + hidden_dim: usize, + num_heads: usize, + + /// Threshold for full attention (criticality > threshold) + gate_threshold: f32, + + /// Min-cut engine for criticality computation + mincut_engine: Arc, + + /// Projection weights + w_query: Array2, + w_key: Array2, + w_value: Array2, + + /// Gate projection (learns to predict criticality) + w_gate: Array2, +} + +impl MinCutGatedAttention { + pub fn forward(&self, query_node: &DagNode, ctx: &DagContext) -> AttentionOutput { + // 1. Get or compute criticalities + let criticalities = self.get_criticalities(ctx); + + // 2. Project query + let q = self.project_query(&query_node.embedding); + let scale = (self.hidden_dim as f32).sqrt(); + + // 3. Compute gated attention + let ancestors = self.get_ancestors(query_node.id, ctx); + let mut scores = Vec::with_capacity(ctx.nodes.len()); + let mut gates = Vec::with_capacity(ctx.nodes.len()); + + for (i, node) in ctx.nodes.iter().enumerate() { + if ancestors.contains(&node.id) { + let k = self.project_key(&node.embedding); + let base_score = dot(&q, &k) / scale; + + // Compute gate based on criticality + let criticality = criticalities.get(&node.id).unwrap_or(&0.0); + let gate = if *criticality > self.gate_threshold { + 1.0 // Full attention for critical nodes + } else { + criticality / self.gate_threshold // Scaled attention + }; + + scores.push(base_score); + gates.push(gate); + } else { + scores.push(f32::NEG_INFINITY); + gates.push(0.0); + } + } + + // 4. Apply gates before softmax + let gated_scores: Vec = scores.iter() + .zip(gates.iter()) + .map(|(s, g)| if *s > f32::NEG_INFINITY { s * g } else { *s }) + .collect(); + + let weights = softmax(&gated_scores); + let aggregated = self.aggregate_values(&weights, ctx); + + AttentionOutput { + weights, + aggregated, + metadata: AttentionMetadata::with_criticalities(criticalities), + } + } + + fn get_criticalities(&self, ctx: &DagContext) -> HashMap { + // Use precomputed if available + if let Some(ref crit) = ctx.criticalities { + return crit.clone(); + } + + // Compute via min-cut engine + let mut criticalities = HashMap::new(); + let global_cut = self.mincut_engine.query(); + + for node in &ctx.nodes { + // LocalKCut query around this node + let local_query = LocalKCutQuery { + seed_vertices: vec![node.id], + budget_k: global_cut, + radius: 3, + }; + + match self.mincut_engine.local_query(local_query) { + LocalKCutResult::Found { cut_value, .. } => { + let criticality = (global_cut - cut_value) as f32 / global_cut as f32; + criticalities.insert(node.id, criticality); + } + LocalKCutResult::NoneInLocality => { + criticalities.insert(node.id, 0.0); + } + } + } + + criticalities + } +} +``` + +### Configuration + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `gate_threshold` | 0.5 | Criticality threshold for full attention | + +### Complexity +- Time: O(n^0.12 + n·k) - subpolynomial min-cut + attention +- Space: O(n) for criticality map + +### SQL Interface + +```sql +SELECT ruvector_attention_mincut_gated( + query_embedding, + ancestor_embeddings, + ruvector_mincut_criticality(dag_id), -- precomputed criticalities + 0.5 -- threshold +) AS attention_weights; +``` + +--- + +## 5. Hierarchical Lorentz Attention + +### Purpose +Uses hyperbolic geometry (Lorentz hyperboloid) to naturally represent DAG hierarchy. Deeper nodes embed further from origin. + +### Algorithm + +```rust +pub struct HierarchicalLorentzAttention { + hidden_dim: usize, + num_heads: usize, + + /// Curvature of hyperbolic space (negative) + curvature: f32, + + /// Temperature for attention softmax + temperature: f32, + + /// Lorentz model for hyperbolic operations + lorentz: LorentzModel, + + /// Projection to hyperbolic space + w_to_hyperbolic: Array2, +} + +impl HierarchicalLorentzAttention { + pub fn forward(&self, query_node: &DagNode, ctx: &DagContext) -> AttentionOutput { + let query_depth = ctx.depths[&query_node.id]; + + // 1. Map query to hyperbolic space at its depth + let query_hyper = self.to_lorentz( + &query_node.embedding, + query_depth, + ); + + // 2. Compute Lorentz attention scores + let ancestors = self.get_ancestors(query_node.id, ctx); + let mut scores = Vec::with_capacity(ctx.nodes.len()); + + for (i, node) in ctx.nodes.iter().enumerate() { + if ancestors.contains(&node.id) { + let node_depth = ctx.depths[&node.id]; + let node_hyper = self.to_lorentz(&node.embedding, node_depth); + + // Busemann function for O(d) hierarchy scoring + let score = self.lorentz.busemann_score(&query_hyper, &node_hyper); + scores.push(score / self.temperature); + } else { + scores.push(f32::NEG_INFINITY); + } + } + + // 3. Hyperbolic softmax + let weights = self.lorentz.hyperbolic_softmax(&scores); + + // 4. Einstein midpoint aggregation (closed-form, no iteration) + let aggregated = self.einstein_midpoint_aggregation(&weights, ctx); + + AttentionOutput { + weights, + aggregated, + metadata: AttentionMetadata::new(DagAttentionType::HierarchicalLorentz), + } + } + + fn to_lorentz(&self, embedding: &[f32], depth: usize) -> LorentzPoint { + // Project to hyperbolic space + let projected = self.w_to_hyperbolic.dot(&Array1::from_vec(embedding.to_vec())); + + // Map depth to hyperbolic radius + let radius = (depth as f32 + 1.0).ln() / (-self.curvature).sqrt(); + + self.lorentz.exp_map_at_origin(&projected.to_vec(), radius) + } + + fn einstein_midpoint_aggregation( + &self, + weights: &[f32], + ctx: &DagContext, + ) -> Vec { + // Closed-form weighted centroid in Lorentz model + // Much faster than iterative Fréchet mean + + let mut numerator = vec![0.0; self.hidden_dim + 1]; + let mut denominator = 0.0; + + for (i, node) in ctx.nodes.iter().enumerate() { + if weights[i] > 1e-8 { + let depth = ctx.depths[&node.id]; + let hyper = self.to_lorentz(&node.embedding, depth); + + // Lorentz factor + let gamma = hyper.lorentz_factor(); + let weight = weights[i] * gamma; + + for (j, &coord) in hyper.coordinates().iter().enumerate() { + numerator[j] += weight * coord; + } + denominator += weight; + } + } + + // Normalize and project back to Euclidean + let midpoint: Vec = numerator.iter() + .map(|x| x / denominator) + .collect(); + + self.lorentz.log_map_to_euclidean(&midpoint) + } +} +``` + +### Configuration + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `curvature` | -1.0 | Hyperbolic space curvature | +| `temperature` | 1.0 | Softmax temperature | + +### Benefits +- 5-10x faster than Poincaré attention +- Numerically stable (no tanh clipping) +- Natural hierarchy representation + +### SQL Interface + +```sql +SELECT ruvector_attention_hierarchical_lorentz( + query_embedding, + ancestor_embeddings, + ancestor_depths, -- INT[]: DAG depths + -1.0, -- curvature + 1.0 -- temperature +) AS attention_weights; +``` + +--- + +## 6. Parallel Branch Attention + +### Purpose +Coordinates across parallel DAG branches. Essential for parallel query execution where branches need to share information. + +### Algorithm + +```rust +pub struct ParallelBranchAttention { + hidden_dim: usize, + num_heads: usize, + + /// Weight for cross-branch attention + cross_branch_weight: f32, + + /// Boost for common ancestors (synchronization points) + common_ancestor_boost: f32, + + /// Projection weights + w_query: Array2, + w_key: Array2, + w_value: Array2, + + /// Cross-attention projection + w_cross: Array2, +} + +impl ParallelBranchAttention { + pub fn forward(&self, query_node: &DagNode, ctx: &DagContext) -> AttentionOutput { + let query_depth = ctx.depths[&query_node.id]; + + // 1. Find parallel branches (same depth, different lineage) + let parallel_nodes = self.find_parallel_nodes(query_node.id, query_depth, ctx); + + // 2. Find common ancestors (synchronization points) + let common_ancestors = self.find_common_ancestors( + query_node.id, + ¶llel_nodes, + ctx, + ); + + // 3. Project query + let q = self.project_query(&query_node.embedding); + let scale = (self.hidden_dim as f32).sqrt(); + + // 4. Compute multi-source attention + let ancestors = self.get_ancestors(query_node.id, ctx); + let mut scores = Vec::new(); + let mut node_types = Vec::new(); // Track source type + + // Own ancestors + for node in &ctx.nodes { + if ancestors.contains(&node.id) { + let k = self.project_key(&node.embedding); + let mut score = dot(&q, &k) / scale; + + // Boost common ancestors + if common_ancestors.contains(&node.id) { + score += self.common_ancestor_boost; + } + + scores.push(score); + node_types.push(NodeType::Ancestor); + } + } + + // Parallel branch nodes (cross-attention) + for ¶llel_id in ¶llel_nodes { + let parallel_node = ctx.get_node(parallel_id); + let k = self.project_cross(¶llel_node.embedding); + let score = dot(&q, &k) / scale * self.cross_branch_weight; + + scores.push(score); + node_types.push(NodeType::ParallelBranch); + } + + // 5. Softmax over all sources + let weights = softmax(&scores); + + // 6. Aggregation (separate paths for different types) + let aggregated = self.multi_source_aggregate(&weights, &node_types, ctx); + + AttentionOutput { + weights, + aggregated, + metadata: AttentionMetadata::with_parallel_info(parallel_nodes, common_ancestors), + } + } + + fn find_parallel_nodes( + &self, + query_id: NodeId, + query_depth: usize, + ctx: &DagContext, + ) -> Vec { + let query_ancestors = self.get_ancestors(query_id, ctx); + + ctx.nodes.iter() + .filter(|n| { + n.id != query_id && + ctx.depths[&n.id] == query_depth && + !query_ancestors.contains(&n.id) // Not an ancestor + }) + .map(|n| n.id) + .collect() + } + + fn find_common_ancestors( + &self, + query_id: NodeId, + parallel_ids: &[NodeId], + ctx: &DagContext, + ) -> HashSet { + if parallel_ids.is_empty() { + return HashSet::new(); + } + + // Start with query's ancestors + let mut common = self.get_ancestors(query_id, ctx); + + // Intersect with each parallel node's ancestors + for ¶llel_id in parallel_ids { + let parallel_ancestors = self.get_ancestors(parallel_id, ctx); + common = common.intersection(¶llel_ancestors).cloned().collect(); + } + + common + } +} +``` + +### Configuration + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `cross_branch_weight` | 0.5 | Weight for cross-branch attention | +| `common_ancestor_boost` | 2.0 | Boost for synchronization points | + +### SQL Interface + +```sql +SELECT ruvector_attention_parallel_branch( + query_embedding, + ancestor_embeddings, + parallel_embeddings, -- VECTOR[]: parallel branch embeddings + common_ancestor_mask, -- BOOLEAN[]: common ancestor indicators + 0.5, -- cross_weight + 2.0 -- common_ancestor_boost +) AS attention_weights; +``` + +--- + +## 7. Temporal BTSP Attention + +### Purpose +Combines DAG structure with temporal spike patterns. Uses BTSP (Behavioral Timescale Synaptic Plasticity) for one-shot learning of time-correlated query patterns. + +### Algorithm + +```rust +pub struct TemporalBTSPAttention { + hidden_dim: usize, + num_heads: usize, + + /// Coincidence window for temporal grouping (ms) + coincidence_window_ms: f32, + + /// Boost for temporally coincident nodes + coincidence_boost: f32, + + /// BTSP memory for one-shot pattern recall + btsp_memory: BTSPLayer, + + /// Projection weights + w_query: Array2, + w_key: Array2, + w_value: Array2, +} + +impl TemporalBTSPAttention { + pub fn forward( + &self, + query_node: &DagNode, + ctx: &DagContext, + ) -> AttentionOutput { + let timestamps = ctx.timestamps.as_ref() + .expect("Temporal attention requires timestamps"); + + let query_time = timestamps[&query_node.id]; + + // 1. Try BTSP recall first (one-shot memory) + if let Some(recalled) = self.btsp_memory.recall(&query_node.embedding) { + return AttentionOutput { + weights: recalled.weights, + aggregated: recalled.aggregated, + metadata: AttentionMetadata::btsp_recalled(), + }; + } + + // 2. Find temporally coincident nodes + let coincident: HashSet = ctx.nodes.iter() + .filter(|n| { + let t = timestamps[&n.id]; + (query_time - t).abs() < self.coincidence_window_ms as f64 + }) + .map(|n| n.id) + .collect(); + + // 3. Compute attention with temporal boost + let ancestors = self.get_ancestors(query_node.id, ctx); + let q = self.project_query(&query_node.embedding); + let scale = (self.hidden_dim as f32).sqrt(); + + let mut scores = Vec::with_capacity(ctx.nodes.len()); + + for node in &ctx.nodes { + if ancestors.contains(&node.id) { + let k = self.project_key(&node.embedding); + let mut score = dot(&q, &k) / scale; + + // Boost temporally coincident nodes + if coincident.contains(&node.id) { + score *= self.coincidence_boost; + } + + scores.push(score); + } else { + scores.push(f32::NEG_INFINITY); + } + } + + let weights = softmax(&scores); + let aggregated = self.aggregate_values(&weights, ctx); + + // 4. One-shot learning via BTSP plateau + if self.should_learn(&weights) { + self.btsp_memory.associate( + &query_node.embedding, + &weights, + &aggregated, + ); + } + + AttentionOutput { + weights, + aggregated, + metadata: AttentionMetadata::with_temporal_info(coincident), + } + } + + fn should_learn(&self, weights: &[f32]) -> bool { + // Learn if attention is confident (low entropy) + let entropy: f32 = weights.iter() + .filter(|&&w| w > 1e-8) + .map(|&w| -w * w.ln()) + .sum(); + + let max_entropy = (weights.len() as f32).ln(); + let normalized_entropy = entropy / max_entropy; + + normalized_entropy < 0.5 // Confident attention pattern + } +} +``` + +### Configuration + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `coincidence_window_ms` | 50.0 | Temporal coincidence window | +| `coincidence_boost` | 1.5 | Boost for coincident nodes | + +### BTSP Memory Structure + +```rust +pub struct BTSPLayer { + /// Synaptic weights: pattern → attention + synapses: Vec, + + /// Number of stored patterns + pattern_count: usize, + + /// Maximum patterns before consolidation + max_patterns: usize, + + /// Similarity threshold for recall + recall_threshold: f32, +} + +pub struct BTSPSynapse { + /// Input pattern embedding + pattern: Vec, + + /// Learned attention weights + attention_weights: Vec, + + /// Learned aggregated output + aggregated: Vec, + + /// Confidence score + confidence: f32, + + /// Usage count + usage_count: usize, +} +``` + +### SQL Interface + +```sql +SELECT ruvector_attention_temporal_btsp( + query_embedding, + ancestor_embeddings, + ancestor_timestamps, -- FLOAT[]: timestamps in ms + 50.0, -- coincidence_window_ms + 1.5 -- coincidence_boost +) AS attention_weights; +``` + +--- + +## Ensemble Attention + +### Purpose +Combines multiple attention types for robust performance. + +### Algorithm + +```rust +pub struct EnsembleAttention { + /// Component attention mechanisms + components: Vec>, + + /// Combination weights (learned or fixed) + weights: Vec, + + /// Weight learning mode + weight_mode: WeightMode, +} + +pub enum WeightMode { + /// Fixed weights + Fixed, + + /// Learn weights via gradient descent + Learned, + + /// Adaptive based on query pattern + Adaptive(AdaptiveWeightSelector), +} + +impl EnsembleAttention { + pub fn forward(&self, query_node: &DagNode, ctx: &DagContext) -> AttentionOutput { + // Get weights for this query + let weights = match &self.weight_mode { + WeightMode::Fixed => self.weights.clone(), + WeightMode::Learned => self.weights.clone(), + WeightMode::Adaptive(selector) => { + selector.select_weights(&query_node.embedding) + } + }; + + // Compute attention from each component + let outputs: Vec = self.components.iter() + .map(|c| c.forward(query_node, ctx)) + .collect(); + + // Weighted combination + let n = outputs[0].weights.len(); + let mut combined_weights = vec![0.0; n]; + let mut combined_aggregated = vec![0.0; outputs[0].aggregated.len()]; + + for (i, output) in outputs.iter().enumerate() { + let w = weights[i]; + for (j, &ow) in output.weights.iter().enumerate() { + combined_weights[j] += w * ow; + } + for (j, &oa) in output.aggregated.iter().enumerate() { + combined_aggregated[j] += w * oa; + } + } + + // Renormalize weights + let sum: f32 = combined_weights.iter().sum(); + if sum > 0.0 { + for w in &mut combined_weights { + *w /= sum; + } + } + + AttentionOutput { + weights: combined_weights, + aggregated: combined_aggregated, + metadata: AttentionMetadata::ensemble( + self.components.iter().map(|c| c.attention_type()).collect() + ), + } + } +} +``` + +### SQL Interface + +```sql +SELECT ruvector_attention_ensemble( + query_embedding, + ancestor_embeddings, + ARRAY['topological', 'critical_path', 'mincut_gated'], -- types + ARRAY[0.4, 0.3, 0.3]::FLOAT[] -- weights (optional) +) AS attention_weights; +``` + +--- + +## Attention Selector (UCB Bandit) + +### Purpose +Automatically selects the best attention type for each query pattern. + +### Algorithm + +```rust +pub struct AttentionSelector { + /// Performance history per (pattern_type, attention_type) + history: DashMap<(PatternTypeId, DagAttentionType), PerformanceStats>, + + /// UCB exploration coefficient + ucb_c: f32, + + /// Exploration probability (epsilon-greedy) + epsilon: f32, + + /// Pattern type classifier + pattern_classifier: PatternClassifier, +} + +impl AttentionSelector { + pub fn select(&self, query_embedding: &[f32]) -> DagAttentionType { + // Classify query pattern + let pattern_type = self.pattern_classifier.classify(query_embedding); + + // Epsilon-greedy exploration + if rand::random::() < self.epsilon { + return DagAttentionType::random(); + } + + // UCB selection + let total_trials: usize = self.history.iter() + .filter(|e| e.key().0 == pattern_type) + .map(|e| e.value().trials) + .sum(); + + let mut best_ucb = f32::NEG_INFINITY; + let mut best_type = DagAttentionType::Topological; + + for attention_type in DagAttentionType::all() { + let key = (pattern_type, attention_type.clone()); + + let (mean_reward, trials) = self.history.get(&key) + .map(|s| (s.mean_reward(), s.trials)) + .unwrap_or((0.5, 1)); // Optimistic initialization + + // UCB formula + let exploration = self.ucb_c * + ((total_trials as f32).ln() / trials as f32).sqrt(); + let ucb = mean_reward + exploration; + + if ucb > best_ucb { + best_ucb = ucb; + best_type = attention_type; + } + } + + best_type + } + + pub fn update( + &self, + query_embedding: &[f32], + attention_type: DagAttentionType, + reward: f32, + ) { + let pattern_type = self.pattern_classifier.classify(query_embedding); + let key = (pattern_type, attention_type); + + self.history.entry(key) + .or_insert_with(PerformanceStats::new) + .record(reward); + } +} +``` + +### Configuration + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `ucb_c` | 1.414 | UCB exploration coefficient (√2) | +| `epsilon` | 0.1 | Random exploration probability | + +--- + +## Performance Summary + +| Attention Type | Time Complexity | Space Complexity | Best For | +|----------------|-----------------|------------------|----------| +| Topological | O(n·k) | O(n) | Causal consistency | +| Causal Cone | O(n·d) | O(n) | Distance-weighted | +| Critical Path | O(n + crit_len) | O(n) | Bottleneck focus | +| MinCut Gated | O(n^0.12 + n·k) | O(n) | Bottleneck detection | +| Hierarchical Lorentz | O(n·d) | O(n·d) | Deep hierarchies | +| Parallel Branch | O(n·b) | O(n) | Parallel execution | +| Temporal BTSP | O(n·w) | O(patterns) | Repeated queries | +| Ensemble | O(e·n·k) | O(e·n) | Robust performance | + +Where: +- n = number of nodes +- k = average ancestors +- d = DAG depth +- b = parallel branches +- w = coincidence window +- e = ensemble components diff --git a/docs/dag/03-SONA-INTEGRATION.md b/docs/dag/03-SONA-INTEGRATION.md new file mode 100644 index 000000000..5ad3a2bfe --- /dev/null +++ b/docs/dag/03-SONA-INTEGRATION.md @@ -0,0 +1,1009 @@ +# SONA Integration Specification + +## Overview + +SONA (Self-Optimizing Neural Architecture) provides the core learning infrastructure for the Neural DAG system. This document specifies how SONA integrates with RuVector-Postgres for continuous query optimization. + +## SONA Architecture Review + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ SONA ENGINE │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ INSTANT LOOP (<100μs per query) │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ MicroLoRA │ │ Trajectory │ │ Auto- │ │ │ +│ │ │ (rank 1-2) │ │ Buffer │ │ Flush │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ (hourly) │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ BACKGROUND LOOP (1-5s per cycle) │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │ +│ │ │ K-means │ │ Pattern │ │ EWC++ │ │ BaseLoRA │ │ │ +│ │ │ Clustering │ │ Extraction │ │ Constraints │ │ (rank 8) │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ REASONING BANK │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ Pattern │ │ Similarity │ │ Eviction │ │ │ +│ │ │ Storage │ │ Search │ │ Policy │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## Integration Design + +### DagSonaEngine Wrapper + +```rust +/// Main integration point between DAG system and SONA +pub struct DagSonaEngine { + /// Core SONA engine (from sona crate) + inner: SonaEngine, + + /// DAG-specific configuration + config: DagSonaConfig, + + /// DAG trajectory buffer (specialized for query plans) + dag_trajectory_buffer: Arc, + + /// DAG reasoning bank (pattern storage) + dag_reasoning_bank: Arc>, + + /// Attention selector integration + attention_selector: Arc, + + /// MinCut engine integration (optional) + mincut_engine: Option>, + + /// Background worker handle + worker_handle: Option>, + + /// Metrics collector + metrics: Arc, +} + +impl DagSonaEngine { + /// Create new engine for a table + pub fn new(table_name: &str, config: DagSonaConfig) -> Result { + let inner = SonaEngine::new(SonaConfig { + hidden_dim: config.hidden_dim, + micro_lora_rank: config.micro_lora_rank, + base_lora_rank: config.base_lora_rank, + ewc_lambda: config.ewc_lambda, + ..Default::default() + })?; + + Ok(Self { + inner, + config, + dag_trajectory_buffer: Arc::new(DagTrajectoryBuffer::new( + config.max_trajectories + )), + dag_reasoning_bank: Arc::new(RwLock::new(DagReasoningBank::new( + config.max_patterns + ))), + attention_selector: Arc::new(AttentionSelector::new( + config.ucb_c, + config.epsilon, + )), + mincut_engine: None, + worker_handle: None, + metrics: Arc::new(DagSonaMetrics::new()), + }) + } + + /// Enable MinCut integration + pub fn with_mincut(mut self, engine: Arc) -> Self { + self.mincut_engine = Some(engine); + self + } + + /// Start background learning worker + pub fn start_background_worker(&mut self) -> Result<(), SonaError> { + let engine = self.clone_for_worker(); + let interval = self.config.background_interval; + + let handle = std::thread::spawn(move || { + loop { + std::thread::sleep(interval); + + if let Err(e) = engine.run_background_cycle() { + log::error!("Background learning cycle failed: {}", e); + } + } + }); + + self.worker_handle = Some(handle); + Ok(()) + } +} +``` + +### DAG-Specific Trajectory + +```rust +/// Trajectory specialized for DAG query plans +#[derive(Clone, Debug)] +pub struct DagTrajectory { + /// Unique trajectory ID + pub id: u64, + + /// Query plan embedding (256-dim) + pub plan_embedding: Vec, + + /// Operator-level embeddings + pub operator_embeddings: Vec>, + + /// Attention weights used + pub attention_weights: Vec>, + + /// Attention type used + pub attention_type: DagAttentionType, + + /// Execution parameters + pub params: ExecutionParams, + + /// Execution metrics + pub metrics: ExecutionMetrics, + + /// Quality score (computed) + pub quality: f32, + + /// Timestamp + pub timestamp: SystemTime, +} + +#[derive(Clone, Debug)] +pub struct ExecutionParams { + /// HNSW ef_search if applicable + pub ef_search: Option, + + /// IVFFlat probes if applicable + pub probes: Option, + + /// Parallelism level + pub parallelism: usize, + + /// Selected attention type + pub attention_type: DagAttentionType, + + /// Custom parameters + pub custom: HashMap, +} + +#[derive(Clone, Debug)] +pub struct ExecutionMetrics { + /// Total execution time (microseconds) + pub latency_us: u64, + + /// Planning time (microseconds) + pub planning_us: u64, + + /// Execution time (microseconds) + pub execution_us: u64, + + /// Rows processed + pub rows_processed: u64, + + /// Memory used (bytes) + pub memory_bytes: u64, + + /// Cache hit rate + pub cache_hit_rate: f32, + + /// MinCut criticality (if computed) + pub max_criticality: Option, +} + +impl DagTrajectory { + /// Compute quality score from metrics + pub fn compute_quality(&mut self) { + // Multi-objective quality function + let latency_score = 1.0 / (1.0 + self.metrics.latency_us as f32 / 1000.0); + let memory_score = 1.0 / (1.0 + self.metrics.memory_bytes as f32 / 1_000_000.0); + let cache_score = self.metrics.cache_hit_rate; + + // Weighted combination + self.quality = 0.5 * latency_score + 0.3 * memory_score + 0.2 * cache_score; + } +} +``` + +### DAG Trajectory Buffer + +```rust +/// Lock-free buffer for DAG trajectories +pub struct DagTrajectoryBuffer { + /// Lock-free queue + buffer: ArrayQueue, + + /// Maximum capacity + capacity: usize, + + /// Dropped count (for monitoring) + dropped: AtomicU64, + + /// Total seen (for statistics) + total_seen: AtomicU64, +} + +impl DagTrajectoryBuffer { + pub fn new(capacity: usize) -> Self { + Self { + buffer: ArrayQueue::new(capacity), + capacity, + dropped: AtomicU64::new(0), + total_seen: AtomicU64::new(0), + } + } + + /// Record a trajectory (non-blocking) + pub fn record(&self, trajectory: DagTrajectory) -> bool { + self.total_seen.fetch_add(1, Ordering::Relaxed); + + match self.buffer.push(trajectory) { + Ok(()) => true, + Err(_) => { + self.dropped.fetch_add(1, Ordering::Relaxed); + false + } + } + } + + /// Drain all trajectories for learning + pub fn drain(&self) -> Vec { + let mut trajectories = Vec::with_capacity(self.capacity); + while let Some(t) = self.buffer.pop() { + trajectories.push(t); + } + trajectories + } + + /// Get buffer statistics + pub fn stats(&self) -> BufferStats { + BufferStats { + current_size: self.buffer.len(), + capacity: self.capacity, + total_seen: self.total_seen.load(Ordering::Relaxed), + dropped: self.dropped.load(Ordering::Relaxed), + } + } +} +``` + +### DAG Reasoning Bank + +```rust +/// Pattern storage specialized for DAG query plans +pub struct DagReasoningBank { + /// Learned patterns + patterns: DashMap, + + /// Pattern index for similarity search + pattern_index: Vec<(Vec, PatternId)>, + + /// Maximum patterns + max_patterns: usize, + + /// Quality threshold for storing + quality_threshold: f32, + + /// Pattern ID generator + next_id: AtomicU64, +} + +/// Learned pattern for DAG queries +#[derive(Clone, Debug)] +pub struct LearnedDagPattern { + /// Pattern ID + pub id: PatternId, + + /// Centroid embedding (cluster center) + pub centroid: Vec, + + /// Optimal execution parameters + pub optimal_params: ExecutionParams, + + /// Optimal attention type + pub optimal_attention: DagAttentionType, + + /// Confidence score (0.0 - 1.0) + pub confidence: f32, + + /// Sample count (trajectories in this cluster) + pub sample_count: usize, + + /// Average metrics + pub avg_metrics: AverageMetrics, + + /// Last updated + pub updated_at: SystemTime, +} + +#[derive(Clone, Debug)] +pub struct AverageMetrics { + pub latency_us: f64, + pub memory_bytes: f64, + pub quality: f64, +} + +impl DagReasoningBank { + pub fn new(max_patterns: usize) -> Self { + Self { + patterns: DashMap::new(), + pattern_index: Vec::new(), + max_patterns, + quality_threshold: 0.3, + next_id: AtomicU64::new(1), + } + } + + /// Find k most similar patterns + pub fn find_similar(&self, query: &[f32], k: usize) -> Vec<&LearnedDagPattern> { + let mut similarities: Vec<(PatternId, f32)> = self.pattern_index.iter() + .map(|(centroid, id)| (*id, cosine_similarity(query, centroid))) + .collect(); + + similarities.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(Ordering::Equal)); + + similarities.into_iter() + .take(k) + .filter_map(|(id, _)| self.patterns.get(&id).map(|r| r.value())) + .collect() + } + + /// Store a new pattern + pub fn store(&self, pattern: LearnedDagPattern) -> PatternId { + // Check capacity and evict if needed + if self.patterns.len() >= self.max_patterns { + self.evict_lowest_confidence(); + } + + let id = pattern.id; + self.patterns.insert(id, pattern.clone()); + self.pattern_index.push((pattern.centroid.clone(), id)); + + id + } + + /// Consolidate similar patterns + pub fn consolidate(&mut self, similarity_threshold: f32) { + let mut to_merge: Vec<(PatternId, PatternId)> = Vec::new(); + + // Find pairs to merge + for i in 0..self.pattern_index.len() { + for j in (i + 1)..self.pattern_index.len() { + let sim = cosine_similarity( + &self.pattern_index[i].0, + &self.pattern_index[j].0, + ); + if sim > similarity_threshold { + to_merge.push((self.pattern_index[i].1, self.pattern_index[j].1)); + } + } + } + + // Merge patterns + for (keep_id, remove_id) in to_merge { + if let (Some(keep), Some(remove)) = ( + self.patterns.get(&keep_id), + self.patterns.get(&remove_id), + ) { + // Merge into keep (weighted average) + let total_samples = keep.sample_count + remove.sample_count; + let keep_weight = keep.sample_count as f32 / total_samples as f32; + let remove_weight = remove.sample_count as f32 / total_samples as f32; + + let merged_centroid: Vec = keep.centroid.iter() + .zip(remove.centroid.iter()) + .map(|(a, b)| a * keep_weight + b * remove_weight) + .collect(); + + drop(keep); + drop(remove); + + if let Some(mut keep) = self.patterns.get_mut(&keep_id) { + keep.centroid = merged_centroid; + keep.sample_count = total_samples; + keep.confidence = (keep.confidence + self.patterns.get(&remove_id) + .map(|r| r.confidence).unwrap_or(0.0)) / 2.0; + } + + self.patterns.remove(&remove_id); + } + } + + // Rebuild index + self.rebuild_index(); + } + + fn evict_lowest_confidence(&self) { + if let Some(min_entry) = self.patterns.iter() + .min_by(|a, b| a.confidence.partial_cmp(&b.confidence).unwrap_or(Ordering::Equal)) + { + let id = *min_entry.key(); + drop(min_entry); + self.patterns.remove(&id); + } + } + + fn rebuild_index(&mut self) { + self.pattern_index = self.patterns.iter() + .map(|entry| (entry.centroid.clone(), *entry.key())) + .collect(); + } +} +``` + +## Instant Loop Integration + +### Per-Query Flow + +```rust +impl DagSonaEngine { + /// Called before query execution + pub fn pre_query(&self, plan: &mut NeuralDagPlan) -> PreQueryResult { + let start = Instant::now(); + + // 1. Embed the query plan + let plan_embedding = self.embed_plan(plan); + + // 2. Find similar patterns + let similar = { + let bank = self.dag_reasoning_bank.read(); + bank.find_similar(&plan_embedding, 5) + }; + + // 3. Check for high-confidence match + if let Some(best) = similar.first() { + if best.confidence > 0.8 { + // Apply learned configuration + plan.params = best.optimal_params.clone(); + plan.attention_type = best.optimal_attention.clone(); + + self.metrics.pattern_hits.fetch_add(1, Ordering::Relaxed); + + return PreQueryResult::PatternApplied { + pattern_id: best.id, + confidence: best.confidence, + planning_time: start.elapsed(), + }; + } + } + + // 4. No good pattern - use defaults with micro-LoRA adaptation + let adapted_costs = self.inner.apply_micro_lora(&plan_embedding); + plan.learned_costs = Some(adapted_costs); + + // 5. Select attention type via UCB bandit + let attention_type = self.attention_selector.select(&plan_embedding); + plan.attention_type = attention_type; + + self.metrics.pattern_misses.fetch_add(1, Ordering::Relaxed); + + PreQueryResult::DefaultWithAdaptation { + attention_type, + planning_time: start.elapsed(), + } + } + + /// Called after query execution + pub fn post_query(&self, plan: &NeuralDagPlan, metrics: ExecutionMetrics) { + // 1. Build trajectory + let mut trajectory = DagTrajectory { + id: self.generate_trajectory_id(), + plan_embedding: self.embed_plan(plan), + operator_embeddings: plan.operator_embeddings.clone(), + attention_weights: plan.attention_weights.clone(), + attention_type: plan.attention_type.clone(), + params: plan.params.clone(), + metrics: metrics.clone(), + quality: 0.0, + timestamp: SystemTime::now(), + }; + + // 2. Compute quality + trajectory.compute_quality(); + + // 3. Record trajectory (non-blocking) + self.dag_trajectory_buffer.record(trajectory.clone()); + + // 4. Instant learning signal + let signal = LearningSignal::from_dag_trajectory(&trajectory); + self.inner.instant_loop().on_signal(signal); + + // 5. Update attention selector + self.attention_selector.update( + &trajectory.plan_embedding, + trajectory.attention_type.clone(), + trajectory.quality, + ); + + // 6. Update metrics + self.metrics.queries_processed.fetch_add(1, Ordering::Relaxed); + self.metrics.total_latency_us.fetch_add(metrics.latency_us, Ordering::Relaxed); + } + + /// Embed a query plan into vector space + fn embed_plan(&self, plan: &NeuralDagPlan) -> Vec { + // Combine operator embeddings with attention + let mut embedding = vec![0.0; self.config.hidden_dim]; + + for (i, op_emb) in plan.operator_embeddings.iter().enumerate() { + let weight = 1.0 / (i + 1) as f32; // Decay by position + for (j, &val) in op_emb.iter().enumerate() { + if j < embedding.len() { + embedding[j] += weight * val; + } + } + } + + // L2 normalize + let norm: f32 = embedding.iter().map(|x| x * x).sum::().sqrt(); + if norm > 1e-8 { + for x in &mut embedding { + *x /= norm; + } + } + + embedding + } +} +``` + +## Background Loop Integration + +### Learning Cycle + +```rust +impl DagSonaEngine { + /// Run one background learning cycle + pub fn run_background_cycle(&self) -> Result { + let start = Instant::now(); + + // 1. Drain trajectory buffer + let trajectories = self.dag_trajectory_buffer.drain(); + if trajectories.len() < self.config.min_trajectories { + return Ok(BackgroundCycleResult::Skipped { + reason: "Insufficient trajectories".to_string(), + count: trajectories.len(), + }); + } + + // 2. Filter by quality threshold + let quality_trajectories: Vec<_> = trajectories.into_iter() + .filter(|t| t.quality >= self.config.quality_threshold) + .collect(); + + // 3. K-means++ clustering + let patterns = self.extract_patterns(&quality_trajectories)?; + + // 4. Apply EWC++ constraints + let gradients = self.compute_pattern_gradients(&patterns); + let constrained_gradients = self.inner.ewc().apply_constraints(&gradients); + + // 5. Check for task boundary + if self.inner.ewc().detect_task_boundary(&gradients) { + self.inner.ewc_mut().start_new_task(); + log::info!("Detected task boundary, starting new EWC task"); + } + + // 6. Update Fisher information + self.inner.ewc_mut().update_fisher(&constrained_gradients); + + // 7. Update BaseLoRA + self.inner.base_lora_mut().update(&constrained_gradients, self.config.base_lora_lr); + + // 8. Store patterns in ReasoningBank + let stored_count = { + let mut bank = self.dag_reasoning_bank.write(); + let mut count = 0; + for pattern in patterns { + if pattern.confidence >= self.config.pattern_confidence_threshold { + bank.store(pattern); + count += 1; + } + } + count + }; + + // 9. Consolidate similar patterns periodically + if self.should_consolidate() { + let mut bank = self.dag_reasoning_bank.write(); + bank.consolidate(self.config.consolidation_threshold); + } + + let result = BackgroundCycleResult::Completed { + trajectories_processed: quality_trajectories.len(), + patterns_extracted: stored_count, + duration: start.elapsed(), + }; + + self.metrics.background_cycles.fetch_add(1, Ordering::Relaxed); + + Ok(result) + } + + /// K-means++ pattern extraction + fn extract_patterns(&self, trajectories: &[DagTrajectory]) -> Result, SonaError> { + let k = self.config.pattern_clusters.min(trajectories.len()); + if k == 0 { + return Ok(Vec::new()); + } + + // Extract embeddings + let embeddings: Vec> = trajectories.iter() + .map(|t| t.plan_embedding.clone()) + .collect(); + + // K-means++ initialization + let mut centroids = self.kmeans_plusplus_init(&embeddings, k); + + // K-means iterations + for _ in 0..self.config.kmeans_max_iterations { + // Assign points to clusters + let assignments: Vec = embeddings.iter() + .map(|e| self.nearest_centroid(e, ¢roids)) + .collect(); + + // Update centroids + let mut new_centroids = vec![vec![0.0; self.config.hidden_dim]; k]; + let mut counts = vec![0usize; k]; + + for (i, embedding) in embeddings.iter().enumerate() { + let cluster = assignments[i]; + counts[cluster] += 1; + for (j, &val) in embedding.iter().enumerate() { + new_centroids[cluster][j] += val; + } + } + + for (i, centroid) in new_centroids.iter_mut().enumerate() { + if counts[i] > 0 { + for val in centroid.iter_mut() { + *val /= counts[i] as f32; + } + } + } + + // Check convergence + let max_shift: f32 = centroids.iter() + .zip(new_centroids.iter()) + .map(|(old, new)| euclidean_distance(old, new)) + .fold(0.0, f32::max); + + centroids = new_centroids; + + if max_shift < self.config.kmeans_convergence { + break; + } + } + + // Build patterns from clusters + let assignments: Vec = embeddings.iter() + .map(|e| self.nearest_centroid(e, ¢roids)) + .collect(); + + let mut patterns = Vec::with_capacity(k); + + for (cluster_idx, centroid) in centroids.into_iter().enumerate() { + let members: Vec<&DagTrajectory> = trajectories.iter() + .enumerate() + .filter(|(i, _)| assignments[*i] == cluster_idx) + .map(|(_, t)| t) + .collect(); + + if members.is_empty() { + continue; + } + + // Compute optimal parameters (mode of discrete, mean of continuous) + let optimal_attention = self.mode_attention_type(&members); + let optimal_params = self.average_params(&members); + + // Compute average metrics + let avg_metrics = AverageMetrics { + latency_us: members.iter().map(|t| t.metrics.latency_us as f64).sum::() / members.len() as f64, + memory_bytes: members.iter().map(|t| t.metrics.memory_bytes as f64).sum::() / members.len() as f64, + quality: members.iter().map(|t| t.quality as f64).sum::() / members.len() as f64, + }; + + // Confidence based on sample count and quality variance + let quality_variance = self.compute_variance(members.iter().map(|t| t.quality as f64)); + let confidence = (1.0 - quality_variance.sqrt()).max(0.0) * + (1.0 - 1.0 / (members.len() as f32 + 1.0)); + + patterns.push(LearnedDagPattern { + id: self.generate_pattern_id(), + centroid, + optimal_params, + optimal_attention, + confidence, + sample_count: members.len(), + avg_metrics, + updated_at: SystemTime::now(), + }); + } + + Ok(patterns) + } + + fn kmeans_plusplus_init(&self, embeddings: &[Vec], k: usize) -> Vec> { + let mut centroids = Vec::with_capacity(k); + + // First centroid: deterministic (index 0) + centroids.push(embeddings[0].clone()); + + // Remaining centroids: D² weighted + for _ in 1..k { + let distances: Vec = embeddings.iter() + .map(|e| { + centroids.iter() + .map(|c| euclidean_distance(e, c)) + .fold(f32::INFINITY, f32::min) + }) + .collect(); + + let sum: f32 = distances.iter().map(|d| d * d).sum(); + let threshold = rand::random::() * sum; + + let mut cumsum = 0.0; + for (i, &d) in distances.iter().enumerate() { + cumsum += d * d; + if cumsum >= threshold { + centroids.push(embeddings[i].clone()); + break; + } + } + } + + centroids + } +} +``` + +## EWC++ Integration + +### Configuration + +```rust +pub struct EwcConfig { + /// Initial regularization strength + pub lambda: f32, // 2000.0 + + /// Maximum lambda after adaptation + pub max_lambda: f32, // 15000.0 + + /// Minimum lambda + pub min_lambda: f32, // 100.0 + + /// Fisher information EMA decay + pub fisher_decay: f32, // 0.999 + + /// Task boundary detection threshold (z-score) + pub boundary_threshold: f32, // 2.0 + + /// Maximum remembered tasks + pub max_tasks: usize, // 10 + + /// Gradient history size for boundary detection + pub gradient_history_size: usize, // 100 +} +``` + +### Task Boundary Detection + +```rust +impl DagSonaEngine { + /// Detect if query patterns have shifted significantly + fn detect_pattern_shift(&self, recent_trajectories: &[DagTrajectory]) -> bool { + if recent_trajectories.len() < 50 { + return false; + } + + // Compute embedding distribution statistics + let embeddings: Vec<&Vec> = recent_trajectories.iter() + .map(|t| &t.plan_embedding) + .collect(); + + let mean = self.compute_mean_embedding(&embeddings); + let variance = self.compute_embedding_variance(&embeddings, &mean); + + // Compare with historical statistics + let historical_mean = self.get_historical_mean(); + let historical_variance = self.get_historical_variance(); + + // Z-score test + let diff_norm = euclidean_distance(&mean, &historical_mean); + let std = (historical_variance + variance).sqrt() / 2.0; + + let z_score = diff_norm / std; + + z_score > self.config.ewc_boundary_threshold + } +} +``` + +## MicroLoRA Integration + +### Per-Query Adaptation + +```rust +pub struct DagMicroLoRA { + /// Down projection: hidden_dim × rank + down_proj: Vec, + + /// Up projection: rank × hidden_dim + up_proj: Vec, + + /// Rank (1-2 for efficiency) + rank: usize, + + /// Hidden dimension + hidden_dim: usize, + + /// Accumulated gradients + grad_down: Vec, + grad_up: Vec, + + /// Update count for averaging + update_count: usize, + + /// Scale factor: 1.0 / sqrt(rank) + scale: f32, +} + +impl DagMicroLoRA { + /// Apply LoRA to plan embedding + pub fn forward(&self, input: &[f32]) -> Vec { + let mut output = input.to_vec(); + + // Down projection: input → intermediate + let mut intermediate = vec![0.0; self.rank]; + for r in 0..self.rank { + for i in 0..self.hidden_dim { + intermediate[r] += input[i] * self.down_proj[r * self.hidden_dim + i]; + } + } + + // Up projection: intermediate → output delta + for i in 0..self.hidden_dim { + let mut delta = 0.0; + for r in 0..self.rank { + delta += intermediate[r] * self.up_proj[r * self.hidden_dim + i]; + } + output[i] += self.scale * delta; + } + + output + } + + /// Accumulate gradient from learning signal + pub fn accumulate_gradient(&mut self, signal: &LearningSignal) { + // Simplified REINFORCE-style gradient + let quality = signal.quality_score; + + for r in 0..self.rank { + for i in 0..self.hidden_dim { + self.grad_up[r * self.hidden_dim + i] += + signal.gradient_estimate[i] * quality; + } + } + + self.update_count += 1; + } + + /// Apply accumulated gradients + pub fn apply_accumulated(&mut self, learning_rate: f32) { + if self.update_count == 0 { + return; + } + + let scale = learning_rate / self.update_count as f32; + + for (w, g) in self.up_proj.iter_mut().zip(self.grad_up.iter()) { + *w += g * scale; + } + + // Reset accumulators + self.grad_up.fill(0.0); + self.grad_down.fill(0.0); + self.update_count = 0; + } +} +``` + +## Metrics and Monitoring + +```rust +pub struct DagSonaMetrics { + /// Total queries processed + pub queries_processed: AtomicU64, + + /// Pattern cache hits + pub pattern_hits: AtomicU64, + + /// Pattern cache misses + pub pattern_misses: AtomicU64, + + /// Total latency (microseconds) + pub total_latency_us: AtomicU64, + + /// Background cycles completed + pub background_cycles: AtomicU64, + + /// Patterns currently stored + pub patterns_stored: AtomicU64, + + /// Trajectories dropped (buffer full) + pub trajectories_dropped: AtomicU64, + + /// EWC tasks + pub ewc_tasks: AtomicU64, +} + +impl DagSonaMetrics { + pub fn to_json(&self) -> serde_json::Value { + json!({ + "queries_processed": self.queries_processed.load(Ordering::Relaxed), + "pattern_hit_rate": self.hit_rate(), + "avg_latency_us": self.avg_latency(), + "background_cycles": self.background_cycles.load(Ordering::Relaxed), + "patterns_stored": self.patterns_stored.load(Ordering::Relaxed), + "trajectories_dropped": self.trajectories_dropped.load(Ordering::Relaxed), + "ewc_tasks": self.ewc_tasks.load(Ordering::Relaxed), + }) + } + + fn hit_rate(&self) -> f64 { + let hits = self.pattern_hits.load(Ordering::Relaxed) as f64; + let misses = self.pattern_misses.load(Ordering::Relaxed) as f64; + if hits + misses > 0.0 { + hits / (hits + misses) + } else { + 0.0 + } + } + + fn avg_latency(&self) -> f64 { + let total = self.total_latency_us.load(Ordering::Relaxed) as f64; + let count = self.queries_processed.load(Ordering::Relaxed) as f64; + if count > 0.0 { + total / count + } else { + 0.0 + } + } +} +``` + +## Configuration Summary + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `hidden_dim` | 256 | Embedding dimension | +| `micro_lora_rank` | 2 | MicroLoRA rank | +| `micro_lora_lr` | 0.002 | MicroLoRA learning rate | +| `base_lora_rank` | 8 | BaseLoRA rank | +| `base_lora_lr` | 0.001 | BaseLoRA learning rate | +| `max_trajectories` | 10000 | Trajectory buffer size | +| `min_trajectories` | 100 | Min trajectories for learning | +| `pattern_clusters` | 100 | K-means cluster count | +| `quality_threshold` | 0.3 | Min quality for learning | +| `pattern_confidence_threshold` | 0.5 | Min confidence to store | +| `consolidation_threshold` | 0.95 | Similarity for merging | +| `ewc_lambda` | 2000.0 | EWC regularization | +| `ewc_max_lambda` | 15000.0 | Max EWC lambda | +| `ewc_boundary_threshold` | 2.0 | Task boundary z-score | +| `background_interval` | 1 hour | Learning cycle interval | +| `kmeans_max_iterations` | 100 | K-means iterations | +| `kmeans_convergence` | 0.001 | K-means convergence threshold | diff --git a/docs/dag/04-POSTGRES-INTEGRATION.md b/docs/dag/04-POSTGRES-INTEGRATION.md new file mode 100644 index 000000000..545b004cb --- /dev/null +++ b/docs/dag/04-POSTGRES-INTEGRATION.md @@ -0,0 +1,1202 @@ +# PostgreSQL Integration Specification + +## Overview + +This document specifies how the Neural DAG system integrates with PostgreSQL via the pgrx framework, including type definitions, operators, functions, and background workers. + +## Module Structure + +``` +crates/ruvector-postgres/src/dag/ +├── mod.rs # Module root +├── operators.rs # SQL function definitions +├── types.rs # PostgreSQL type mappings +├── hooks.rs # Query execution hooks +├── worker.rs # Background worker +├── gucs.rs # GUC variable definitions +└── state.rs # Per-connection state +``` + +## GUC Variables + +### Definition + +```rust +// crates/ruvector-postgres/src/dag/gucs.rs + +use pgrx::prelude::*; +use std::ffi::CStr; + +// Global enable/disable +static NEURAL_DAG_ENABLED: GucSetting = GucSetting::new(true); + +// Learning parameters +static DAG_LEARNING_RATE: GucSetting = GucSetting::new(0.002); +static DAG_PATTERN_CLUSTERS: GucSetting = GucSetting::new(100); +static DAG_QUALITY_THRESHOLD: GucSetting = GucSetting::new(0.3); +static DAG_MAX_TRAJECTORIES: GucSetting = GucSetting::new(10000); + +// Attention parameters +static DAG_ATTENTION_TYPE: GucSetting<&'static CStr> = + GucSetting::new(unsafe { CStr::from_bytes_with_nul_unchecked(b"auto\0") }); +static DAG_ATTENTION_EXPLORATION: GucSetting = GucSetting::new(0.1); + +// EWC parameters +static DAG_EWC_LAMBDA: GucSetting = GucSetting::new(2000.0); +static DAG_EWC_MAX_LAMBDA: GucSetting = GucSetting::new(15000.0); + +// MinCut parameters +static DAG_MINCUT_ENABLED: GucSetting = GucSetting::new(true); +static DAG_MINCUT_THRESHOLD: GucSetting = GucSetting::new(0.5); + +// Background worker parameters +static DAG_BACKGROUND_INTERVAL_MS: GucSetting = GucSetting::new(3600000); + +pub fn register_gucs() { + GucRegistry::define_bool_guc( + "ruvector.neural_dag_enabled", + "Enable neural DAG optimization", + "When enabled, queries are optimized using learned patterns", + &NEURAL_DAG_ENABLED, + GucContext::Userset, + GucFlags::default(), + ); + + GucRegistry::define_float_guc( + "ruvector.dag_learning_rate", + "Learning rate for DAG optimization", + "Controls how fast the system adapts to new patterns", + &DAG_LEARNING_RATE, + 0.0001, + 1.0, + GucContext::Userset, + GucFlags::default(), + ); + + GucRegistry::define_int_guc( + "ruvector.dag_pattern_clusters", + "Number of pattern clusters", + "K-means clusters for pattern extraction", + &DAG_PATTERN_CLUSTERS, + 10, + 1000, + GucContext::Userset, + GucFlags::default(), + ); + + GucRegistry::define_float_guc( + "ruvector.dag_quality_threshold", + "Minimum quality for learning", + "Trajectories below this quality are not learned", + &DAG_QUALITY_THRESHOLD, + 0.0, + 1.0, + GucContext::Userset, + GucFlags::default(), + ); + + GucRegistry::define_string_guc( + "ruvector.dag_attention_type", + "Default attention type", + "Options: auto, topological, causal_cone, critical_path, mincut_gated, hierarchical_lorentz, parallel_branch, temporal_btsp", + &DAG_ATTENTION_TYPE, + GucContext::Userset, + GucFlags::default(), + ); + + GucRegistry::define_float_guc( + "ruvector.dag_ewc_lambda", + "EWC regularization strength", + "Higher values prevent forgetting more aggressively", + &DAG_EWC_LAMBDA, + 100.0, + 50000.0, + GucContext::Userset, + GucFlags::default(), + ); + + GucRegistry::define_bool_guc( + "ruvector.dag_mincut_enabled", + "Enable MinCut analysis", + "When enabled, bottleneck operators are identified", + &DAG_MINCUT_ENABLED, + GucContext::Userset, + GucFlags::default(), + ); + + GucRegistry::define_int_guc( + "ruvector.dag_background_interval_ms", + "Background learning interval (milliseconds)", + "How often the background learning cycle runs", + &DAG_BACKGROUND_INTERVAL_MS, + 60000, // 1 minute minimum + 86400000, // 24 hours maximum + GucContext::Sighup, + GucFlags::default(), + ); +} + +/// Get current configuration from GUCs +pub fn get_config() -> DagSonaConfig { + DagSonaConfig { + enabled: NEURAL_DAG_ENABLED.get(), + learning_rate: DAG_LEARNING_RATE.get() as f32, + pattern_clusters: DAG_PATTERN_CLUSTERS.get() as usize, + quality_threshold: DAG_QUALITY_THRESHOLD.get() as f32, + max_trajectories: DAG_MAX_TRAJECTORIES.get() as usize, + attention_type: parse_attention_type(DAG_ATTENTION_TYPE.get()), + attention_exploration: DAG_ATTENTION_EXPLORATION.get() as f32, + ewc_lambda: DAG_EWC_LAMBDA.get() as f32, + ewc_max_lambda: DAG_EWC_MAX_LAMBDA.get() as f32, + mincut_enabled: DAG_MINCUT_ENABLED.get(), + mincut_threshold: DAG_MINCUT_THRESHOLD.get() as f32, + background_interval_ms: DAG_BACKGROUND_INTERVAL_MS.get() as u64, + ..Default::default() + } +} +``` + +## State Management + +### Global State + +```rust +// crates/ruvector-postgres/src/dag/state.rs + +use once_cell::sync::Lazy; +use dashmap::DashMap; +use std::sync::Arc; + +/// Global registry of DAG engines per table +static DAG_ENGINES: Lazy>> = + Lazy::new(|| DashMap::new()); + +/// Get or create DAG engine for a table +pub fn get_or_create_engine(table_name: &str) -> Arc { + DAG_ENGINES.entry(table_name.to_string()) + .or_insert_with(|| { + let config = gucs::get_config(); + Arc::new(DagSonaEngine::new(table_name, config).unwrap()) + }) + .clone() +} + +/// Check if neural DAG is enabled for a table +pub fn is_enabled(table_name: &str) -> bool { + DAG_ENGINES.contains_key(table_name) && gucs::get_config().enabled +} + +/// Remove engine (on DROP TABLE or disable) +pub fn remove_engine(table_name: &str) { + DAG_ENGINES.remove(table_name); +} + +/// Get all engine names (for monitoring) +pub fn list_engines() -> Vec { + DAG_ENGINES.iter().map(|e| e.key().clone()).collect() +} +``` + +### Per-Connection State + +```rust +/// Per-connection state for query tracking +pub struct ConnectionState { + /// Current query trajectory being built + current_trajectory: Option, + + /// Query start time + query_start: Option, + + /// Current table being queried + current_table: Option, +} + +thread_local! { + static CONNECTION_STATE: RefCell = RefCell::new(ConnectionState { + current_trajectory: None, + query_start: None, + current_table: None, + }); +} + +impl ConnectionState { + pub fn start_query(&mut self, table_name: &str) { + self.query_start = Some(Instant::now()); + self.current_table = Some(table_name.to_string()); + self.current_trajectory = Some(TrajectoryBuilder::new()); + } + + pub fn end_query(&mut self) -> Option { + let builder = self.current_trajectory.take()?; + let start = self.query_start.take()?; + let _table = self.current_table.take()?; + + Some(builder.build(start.elapsed())) + } +} +``` + +## SQL Function Implementations + +### Enable/Disable Functions + +```rust +// crates/ruvector-postgres/src/dag/operators.rs + +use pgrx::prelude::*; + +/// Enable neural DAG learning for a table +#[pg_extern] +fn ruvector_enable_neural_dag( + table_name: &str, + config: Option, +) -> bool { + let base_config = gucs::get_config(); + + let config = if let Some(pgrx::JsonB(json)) = config { + // Merge JSON config with base config + merge_config(base_config, &json) + } else { + base_config + }; + + // Validate table exists + let table_oid = get_table_oid(table_name); + if table_oid.is_none() { + ereport!( + PgLogLevel::ERROR, + PgSqlErrorCode::ERRCODE_UNDEFINED_TABLE, + format!("Table '{}' does not exist", table_name) + ); + return false; + } + + // Create engine + let engine = DagSonaEngine::new(table_name, config) + .map_err(|e| { + ereport!( + PgLogLevel::ERROR, + PgSqlErrorCode::ERRCODE_INTERNAL_ERROR, + format!("Failed to create DAG engine: {}", e) + ); + }) + .unwrap(); + + // Register engine + DAG_ENGINES.insert(table_name.to_string(), Arc::new(engine)); + + // Start background worker if not running + start_background_worker_if_needed(); + + true +} + +/// Disable neural DAG learning for a table +#[pg_extern] +fn ruvector_disable_neural_dag(table_name: &str) -> bool { + if DAG_ENGINES.remove(table_name).is_some() { + true + } else { + warning!("Neural DAG was not enabled for table '{}'", table_name); + false + } +} + +/// Check if neural DAG is enabled +#[pg_extern] +fn ruvector_neural_dag_enabled(table_name: &str) -> bool { + state::is_enabled(table_name) +} +``` + +### Pattern Functions + +```rust +/// Get learned patterns for a table +#[pg_extern] +fn ruvector_dag_patterns( + table_name: &str, +) -> TableIterator<'static, ( + name!(pattern_id, i64), + name!(centroid, Vec), + name!(attention_type, String), + name!(confidence, f32), + name!(sample_count, i32), + name!(avg_latency_us, f64), + name!(avg_quality, f64), +)> { + let engine = match DAG_ENGINES.get(table_name) { + Some(e) => e.clone(), + None => { + warning!("Neural DAG not enabled for table '{}'", table_name); + return TableIterator::new(vec![].into_iter()); + } + }; + + let patterns: Vec<_> = { + let bank = engine.dag_reasoning_bank.read(); + bank.patterns.iter() + .map(|entry| { + let p = entry.value(); + ( + p.id as i64, + p.centroid.clone(), + format!("{:?}", p.optimal_attention), + p.confidence, + p.sample_count as i32, + p.avg_metrics.latency_us, + p.avg_metrics.quality, + ) + }) + .collect() + }; + + TableIterator::new(patterns.into_iter()) +} + +/// Force pattern extraction +#[pg_extern] +fn ruvector_dag_extract_patterns( + table_name: &str, + num_clusters: Option, +) -> i32 { + let engine = match DAG_ENGINES.get(table_name) { + Some(e) => e.clone(), + None => { + ereport!( + PgLogLevel::ERROR, + PgSqlErrorCode::ERRCODE_INVALID_PARAMETER_VALUE, + format!("Neural DAG not enabled for table '{}'", table_name) + ); + return 0; + } + }; + + // Override cluster count if specified + if let Some(k) = num_clusters { + // Temporary config override + } + + match engine.run_background_cycle() { + Ok(BackgroundCycleResult::Completed { patterns_extracted, .. }) => { + patterns_extracted as i32 + } + Ok(BackgroundCycleResult::Skipped { reason, .. }) => { + warning!("Pattern extraction skipped: {}", reason); + 0 + } + Err(e) => { + ereport!( + PgLogLevel::ERROR, + PgSqlErrorCode::ERRCODE_INTERNAL_ERROR, + format!("Pattern extraction failed: {}", e) + ); + 0 + } + } +} + +/// Consolidate similar patterns +#[pg_extern] +fn ruvector_dag_consolidate_patterns( + table_name: &str, + similarity_threshold: Option, +) -> i32 { + let engine = match DAG_ENGINES.get(table_name) { + Some(e) => e.clone(), + None => { + warning!("Neural DAG not enabled for table '{}'", table_name); + return 0; + } + }; + + let threshold = similarity_threshold.unwrap_or(0.95); + + let before_count = { + let bank = engine.dag_reasoning_bank.read(); + bank.patterns.len() + }; + + { + let mut bank = engine.dag_reasoning_bank.write(); + bank.consolidate(threshold); + } + + let after_count = { + let bank = engine.dag_reasoning_bank.read(); + bank.patterns.len() + }; + + (before_count - after_count) as i32 +} +``` + +### Attention Functions + +```rust +/// Compute topological attention +#[pg_extern(immutable, parallel_safe)] +fn ruvector_attention_topological( + query: Vec, + ancestors: Vec>, + config: Option, +) -> Vec { + let config = parse_attention_config(config); + let attention = TopologicalAttention::new(config); + + let ctx = build_simple_context(&ancestors); + let query_node = DagNode::from_embedding(query); + + match attention.forward(&query_node, &ctx, &config) { + Ok(output) => output.weights, + Err(e) => { + warning!("Attention computation failed: {}", e); + vec![1.0 / ancestors.len() as f32; ancestors.len()] // Uniform fallback + } + } +} + +/// Compute causal cone attention +#[pg_extern(immutable, parallel_safe)] +fn ruvector_attention_causal_cone( + query: Vec, + ancestors: Vec>, + depths: Vec, + decay_rate: f32, + max_depth: i32, +) -> Vec { + let config = AttentionConfig { + hidden_dim: query.len(), + ..Default::default() + }; + + let attention = CausalConeAttention::new( + config.hidden_dim, + 8, // num_heads + decay_rate, + max_depth as usize, + ); + + let ctx = build_context_with_depths(&ancestors, &depths); + let query_node = DagNode::from_embedding(query); + + match attention.forward(&query_node, &ctx, &config) { + Ok(output) => output.weights, + Err(e) => { + warning!("Causal cone attention failed: {}", e); + vec![1.0 / ancestors.len() as f32; ancestors.len()] + } + } +} + +/// Compute critical path attention +#[pg_extern(immutable, parallel_safe)] +fn ruvector_attention_critical_path( + query: Vec, + ancestors: Vec>, + is_critical: Vec, + boost: f32, +) -> Vec { + let config = AttentionConfig { + hidden_dim: query.len(), + ..Default::default() + }; + + let attention = CriticalPathAttention::new( + config.hidden_dim, + 8, + boost, + ); + + let ctx = build_context_with_critical(&ancestors, &is_critical); + let query_node = DagNode::from_embedding(query); + + match attention.forward(&query_node, &ctx, &config) { + Ok(output) => output.weights, + Err(e) => { + warning!("Critical path attention failed: {}", e); + vec![1.0 / ancestors.len() as f32; ancestors.len()] + } + } +} + +/// Compute mincut gated attention +#[pg_extern(immutable, parallel_safe)] +fn ruvector_attention_mincut_gated( + query: Vec, + ancestors: Vec>, + criticalities: Vec, + threshold: f32, +) -> Vec { + let config = AttentionConfig { + hidden_dim: query.len(), + ..Default::default() + }; + + let attention = MinCutGatedAttention::new( + config.hidden_dim, + 8, + threshold, + ); + + let ctx = build_context_with_criticalities(&ancestors, &criticalities); + let query_node = DagNode::from_embedding(query); + + match attention.forward(&query_node, &ctx, &config) { + Ok(output) => output.weights, + Err(e) => { + warning!("MinCut gated attention failed: {}", e); + vec![1.0 / ancestors.len() as f32; ancestors.len()] + } + } +} + +/// Compute hierarchical Lorentz attention +#[pg_extern(immutable, parallel_safe)] +fn ruvector_attention_hierarchical_lorentz( + query: Vec, + ancestors: Vec>, + depths: Vec, + curvature: f32, + temperature: f32, +) -> Vec { + let config = AttentionConfig { + hidden_dim: query.len(), + ..Default::default() + }; + + let attention = HierarchicalLorentzAttention::new( + config.hidden_dim, + 8, + curvature, + temperature, + ); + + let ctx = build_context_with_depths(&ancestors, &depths); + let query_node = DagNode::from_embedding(query); + + match attention.forward(&query_node, &ctx, &config) { + Ok(output) => output.weights, + Err(e) => { + warning!("Hierarchical Lorentz attention failed: {}", e); + vec![1.0 / ancestors.len() as f32; ancestors.len()] + } + } +} + +/// Compute parallel branch attention +#[pg_extern(immutable, parallel_safe)] +fn ruvector_attention_parallel_branch( + query: Vec, + ancestors: Vec>, + parallel_nodes: Vec>, + common_ancestor_mask: Vec, + cross_weight: f32, + common_boost: f32, +) -> Vec { + let config = AttentionConfig { + hidden_dim: query.len(), + ..Default::default() + }; + + let attention = ParallelBranchAttention::new( + config.hidden_dim, + 8, + cross_weight, + common_boost, + ); + + let ctx = build_context_with_parallel( + &ancestors, + ¶llel_nodes, + &common_ancestor_mask, + ); + let query_node = DagNode::from_embedding(query); + + match attention.forward(&query_node, &ctx, &config) { + Ok(output) => output.weights, + Err(e) => { + warning!("Parallel branch attention failed: {}", e); + let total = ancestors.len() + parallel_nodes.len(); + vec![1.0 / total as f32; total] + } + } +} + +/// Compute temporal BTSP attention +#[pg_extern(immutable, parallel_safe)] +fn ruvector_attention_temporal_btsp( + query: Vec, + ancestors: Vec>, + timestamps: Vec, + window_ms: f32, + boost: f32, +) -> Vec { + let config = AttentionConfig { + hidden_dim: query.len(), + ..Default::default() + }; + + let attention = TemporalBTSPAttention::new( + config.hidden_dim, + 8, + window_ms, + boost, + ); + + let ctx = build_context_with_timestamps(&ancestors, ×tamps); + let query_node = DagNode::from_embedding(query); + + match attention.forward(&query_node, &ctx, &config) { + Ok(output) => output.weights, + Err(e) => { + warning!("Temporal BTSP attention failed: {}", e); + vec![1.0 / ancestors.len() as f32; ancestors.len()] + } + } +} + +/// Compute ensemble attention +#[pg_extern(immutable, parallel_safe)] +fn ruvector_attention_ensemble( + query: Vec, + ancestors: Vec>, + attention_types: Vec, + weights: Option>, +) -> Vec { + let config = AttentionConfig { + hidden_dim: query.len(), + ..Default::default() + }; + + let types: Vec = attention_types.iter() + .filter_map(|s| parse_attention_type_str(s)) + .collect(); + + let weights = weights.unwrap_or_else(|| { + vec![1.0 / types.len() as f32; types.len()] + }); + + let attention = EnsembleAttention::new(types, weights); + + let ctx = build_simple_context(&ancestors); + let query_node = DagNode::from_embedding(query); + + match attention.forward(&query_node, &ctx, &config) { + Ok(output) => output.weights, + Err(e) => { + warning!("Ensemble attention failed: {}", e); + vec![1.0 / ancestors.len() as f32; ancestors.len()] + } + } +} +``` + +### MinCut Functions + +```rust +/// Compute mincut criticality for DAG nodes +#[pg_extern] +fn ruvector_dag_mincut_criticality( + table_name: &str, +) -> TableIterator<'static, ( + name!(node_id, i64), + name!(criticality, f32), + name!(is_bottleneck, bool), +)> { + let engine = match DAG_ENGINES.get(table_name) { + Some(e) => e.clone(), + None => { + warning!("Neural DAG not enabled for table '{}'", table_name); + return TableIterator::new(vec![].into_iter()); + } + }; + + let mincut = match &engine.mincut_engine { + Some(m) => m.clone(), + None => { + warning!("MinCut not enabled for table '{}'", table_name); + return TableIterator::new(vec![].into_iter()); + } + }; + + let threshold = gucs::get_config().mincut_threshold; + + // Compute criticalities + let criticalities = mincut.compute_all_criticalities(); + + let results: Vec<_> = criticalities.into_iter() + .map(|(node_id, crit)| { + (node_id as i64, crit, crit > threshold) + }) + .collect(); + + TableIterator::new(results.into_iter()) +} + +/// Analyze DAG bottlenecks +#[pg_extern] +fn ruvector_dag_mincut_analysis( + table_name: &str, +) -> pgrx::JsonB { + let engine = match DAG_ENGINES.get(table_name) { + Some(e) => e.clone(), + None => { + return pgrx::JsonB(json!({ + "error": "Neural DAG not enabled" + })); + } + }; + + let mincut = match &engine.mincut_engine { + Some(m) => m.clone(), + None => { + return pgrx::JsonB(json!({ + "error": "MinCut not enabled" + })); + } + }; + + let global_cut = mincut.query(); + let criticalities = mincut.compute_all_criticalities(); + + let threshold = gucs::get_config().mincut_threshold; + let bottlenecks: Vec<_> = criticalities.iter() + .filter(|(_, c)| *c > threshold) + .map(|(id, c)| json!({ + "node_id": *id, + "criticality": *c, + })) + .collect(); + + pgrx::JsonB(json!({ + "global_mincut": global_cut, + "bottleneck_count": bottlenecks.len(), + "bottlenecks": bottlenecks, + "threshold": threshold, + })) +} +``` + +### Monitoring Functions + +```rust +/// Get learning statistics +#[pg_extern] +fn ruvector_dag_learning_stats(table_name: &str) -> pgrx::JsonB { + let engine = match DAG_ENGINES.get(table_name) { + Some(e) => e.clone(), + None => { + return pgrx::JsonB(json!({ + "enabled": false, + "error": "Neural DAG not enabled" + })); + } + }; + + let metrics = engine.metrics.to_json(); + let buffer_stats = engine.dag_trajectory_buffer.stats(); + let pattern_count = { + let bank = engine.dag_reasoning_bank.read(); + bank.patterns.len() + }; + + pgrx::JsonB(json!({ + "enabled": true, + "queries_processed": metrics["queries_processed"], + "pattern_hit_rate": metrics["pattern_hit_rate"], + "avg_latency_us": metrics["avg_latency_us"], + "patterns_stored": pattern_count, + "trajectories_buffered": buffer_stats.current_size, + "trajectories_dropped": buffer_stats.dropped, + "background_cycles": metrics["background_cycles"], + "ewc_tasks": metrics["ewc_tasks"], + })) +} + +/// Get health report +#[pg_extern] +fn ruvector_dag_health_report(table_name: &str) -> pgrx::JsonB { + let engine = match DAG_ENGINES.get(table_name) { + Some(e) => e.clone(), + None => { + return pgrx::JsonB(json!({ + "healthy": false, + "error": "Neural DAG not enabled" + })); + } + }; + + let metrics = engine.metrics.to_json(); + let buffer_stats = engine.dag_trajectory_buffer.stats(); + + // Health checks + let mut issues = Vec::new(); + + // Check drop rate + let drop_rate = if buffer_stats.total_seen > 0 { + buffer_stats.dropped as f64 / buffer_stats.total_seen as f64 + } else { + 0.0 + }; + if drop_rate > 0.1 { + issues.push(format!("High trajectory drop rate: {:.1}%", drop_rate * 100.0)); + } + + // Check pattern hit rate + let hit_rate = metrics["pattern_hit_rate"].as_f64().unwrap_or(0.0); + if hit_rate < 0.1 && metrics["queries_processed"].as_u64().unwrap_or(0) > 1000 { + issues.push(format!("Low pattern hit rate: {:.1}%", hit_rate * 100.0)); + } + + // Check MinCut bottlenecks + if let Some(ref mincut) = engine.mincut_engine { + let criticalities = mincut.compute_all_criticalities(); + let severe_bottlenecks = criticalities.values() + .filter(|&&c| c > 0.8) + .count(); + if severe_bottlenecks > 0 { + issues.push(format!("{} severe bottlenecks detected", severe_bottlenecks)); + } + } + + pgrx::JsonB(json!({ + "healthy": issues.is_empty(), + "issues": issues, + "metrics": metrics, + "buffer": { + "size": buffer_stats.current_size, + "capacity": buffer_stats.capacity, + "drop_rate": drop_rate, + }, + })) +} + +/// Force a learning cycle +#[pg_extern] +fn ruvector_dag_learn(table_name: &str) -> pgrx::JsonB { + let engine = match DAG_ENGINES.get(table_name) { + Some(e) => e.clone(), + None => { + return pgrx::JsonB(json!({ + "success": false, + "error": "Neural DAG not enabled" + })); + } + }; + + match engine.run_background_cycle() { + Ok(BackgroundCycleResult::Completed { + trajectories_processed, + patterns_extracted, + duration, + }) => { + pgrx::JsonB(json!({ + "success": true, + "trajectories_processed": trajectories_processed, + "patterns_extracted": patterns_extracted, + "duration_ms": duration.as_millis(), + })) + } + Ok(BackgroundCycleResult::Skipped { reason, count }) => { + pgrx::JsonB(json!({ + "success": false, + "skipped": true, + "reason": reason, + "trajectory_count": count, + })) + } + Err(e) => { + pgrx::JsonB(json!({ + "success": false, + "error": e.to_string(), + })) + } + } +} +``` + +## Background Worker + +```rust +// crates/ruvector-postgres/src/dag/worker.rs + +use pgrx::bgworkers::*; +use std::time::Duration; + +#[pg_guard] +pub extern "C" fn dag_background_worker_main(_arg: pg_sys::Datum) { + BackgroundWorker::attach_signal_handlers(SignalWakeFlags::SIGHUP | SignalWakeFlags::SIGTERM); + + log!( + "DAG background worker started with interval {}ms", + gucs::DAG_BACKGROUND_INTERVAL_MS.get() + ); + + while BackgroundWorker::wait_latch(Some(Duration::from_millis( + gucs::DAG_BACKGROUND_INTERVAL_MS.get() as u64 + ))) { + if BackgroundWorker::sighup_received() { + // Reload configuration + log!("DAG background worker received SIGHUP, reloading config"); + } + + // Run learning cycle for all enabled tables + for engine_ref in DAG_ENGINES.iter() { + let table_name = engine_ref.key(); + let engine = engine_ref.value(); + + match engine.run_background_cycle() { + Ok(BackgroundCycleResult::Completed { patterns_extracted, .. }) => { + log!( + "DAG learning cycle completed for '{}': {} patterns", + table_name, + patterns_extracted + ); + } + Ok(BackgroundCycleResult::Skipped { reason, .. }) => { + // Normal - not enough data + } + Err(e) => { + elog!( + WARNING, + "DAG learning cycle failed for '{}': {}", + table_name, + e + ); + } + } + } + } + + log!("DAG background worker shutting down"); +} + +/// Register the background worker +pub fn register_background_worker() { + BackgroundWorkerBuilder::new("ruvector_dag_worker") + .set_function("dag_background_worker_main") + .set_library("ruvector") + .set_argument(0.into()) + .enable_spi_access() + .set_start_time(BgWorkerStartTime::RecoveryFinished) + .set_restart_time(Some(Duration::from_secs(10))) + .load(); +} +``` + +## Query Execution Hooks + +```rust +// crates/ruvector-postgres/src/dag/hooks.rs + +use pgrx::prelude::*; + +/// Hook into query execution for learning +pub fn install_query_hooks() { + // Install planner hook + unsafe { + prev_planner_hook = planner_hook; + planner_hook = Some(dag_planner_hook); + } + + // Install executor hooks + unsafe { + prev_ExecutorStart_hook = ExecutorStart_hook; + ExecutorStart_hook = Some(dag_executor_start_hook); + + prev_ExecutorEnd_hook = ExecutorEnd_hook; + ExecutorEnd_hook = Some(dag_executor_end_hook); + } +} + +/// Planner hook - optimize query plan +#[pg_guard] +unsafe extern "C" fn dag_planner_hook( + parse: *mut pg_sys::Query, + query_string: *const c_char, + cursor_options: c_int, + bound_params: *mut pg_sys::ParamListInfoData, +) -> *mut pg_sys::PlannedStmt { + // Call original planner + let stmt = if let Some(prev) = prev_planner_hook { + prev(parse, query_string, cursor_options, bound_params) + } else { + pg_sys::standard_planner(parse, query_string, cursor_options, bound_params) + }; + + // Check if neural DAG optimization applies + if !gucs::NEURAL_DAG_ENABLED.get() { + return stmt; + } + + // Extract table name from query + if let Some(table_name) = extract_table_name(parse) { + if let Some(engine) = DAG_ENGINES.get(&table_name) { + // Build neural plan wrapper + let mut neural_plan = NeuralDagPlan::from_planned_stmt(stmt); + + // Apply learned optimizations + let _ = engine.pre_query(&mut neural_plan); + + // Store for post-query processing + CONNECTION_STATE.with(|state| { + state.borrow_mut().start_query(&table_name); + }); + } + } + + stmt +} + +/// Executor start hook - record start time +#[pg_guard] +unsafe extern "C" fn dag_executor_start_hook( + query_desc: *mut pg_sys::QueryDesc, + eflags: c_int, +) { + // Call original + if let Some(prev) = prev_ExecutorStart_hook { + prev(query_desc, eflags); + } else { + pg_sys::standard_ExecutorStart(query_desc, eflags); + } + + // Record timing + CONNECTION_STATE.with(|state| { + if let Some(ref mut traj) = state.borrow_mut().current_trajectory { + traj.mark_execution_start(); + } + }); +} + +/// Executor end hook - record trajectory +#[pg_guard] +unsafe extern "C" fn dag_executor_end_hook(query_desc: *mut pg_sys::QueryDesc) { + // Collect metrics before cleanup + let metrics = extract_execution_metrics(query_desc); + + // Call original + if let Some(prev) = prev_ExecutorEnd_hook { + prev(query_desc); + } else { + pg_sys::standard_ExecutorEnd(query_desc); + } + + // Record trajectory + CONNECTION_STATE.with(|state| { + let mut state = state.borrow_mut(); + if let (Some(trajectory), Some(table_name)) = ( + state.end_query(), + state.current_table.as_ref(), + ) { + if let Some(engine) = DAG_ENGINES.get(table_name) { + engine.post_query(&trajectory.plan, metrics); + } + } + }); +} + +fn extract_execution_metrics(query_desc: *mut pg_sys::QueryDesc) -> ExecutionMetrics { + unsafe { + let estate = (*query_desc).estate; + + ExecutionMetrics { + latency_us: 0, // Computed from timestamps + planning_us: 0, + execution_us: 0, + rows_processed: (*estate).es_processed as u64, + memory_bytes: pg_sys::MemoryContextMemAllocated( + (*estate).es_query_cxt, + false, + ) as u64, + cache_hit_rate: 0.0, // Would need buffer stats + max_criticality: None, + } + } +} +``` + +## Extension Initialization + +```rust +// crates/ruvector-postgres/src/lib.rs (additions) + +#[pg_guard] +pub extern "C" fn _PG_init() { + // ... existing initialization ... + + // Register DAG GUCs + dag::gucs::register_gucs(); + + // Install query hooks + dag::hooks::install_query_hooks(); + + // Register background worker + dag::worker::register_background_worker(); + + pgrx::log!("RuVector Neural DAG module initialized"); +} +``` + +## SQL Installation Script + +```sql +-- Extension installation additions + +-- Create schema for DAG objects +CREATE SCHEMA IF NOT EXISTS ruvector_dag; + +-- Pattern storage table (optional persistence) +CREATE TABLE IF NOT EXISTS ruvector_dag.patterns ( + id BIGSERIAL PRIMARY KEY, + table_name TEXT NOT NULL, + pattern_id BIGINT NOT NULL, + centroid ruvector NOT NULL, + attention_type TEXT NOT NULL, + optimal_params JSONB NOT NULL, + confidence FLOAT NOT NULL, + sample_count INT NOT NULL, + avg_latency_us FLOAT NOT NULL, + avg_quality FLOAT NOT NULL, + created_at TIMESTAMPTZ DEFAULT NOW(), + updated_at TIMESTAMPTZ DEFAULT NOW(), + UNIQUE (table_name, pattern_id) +); + +-- Trajectory history table (optional) +CREATE TABLE IF NOT EXISTS ruvector_dag.trajectory_history ( + id BIGSERIAL PRIMARY KEY, + table_name TEXT NOT NULL, + trajectory_id BIGINT NOT NULL, + plan_embedding ruvector NOT NULL, + attention_type TEXT NOT NULL, + latency_us BIGINT NOT NULL, + quality FLOAT NOT NULL, + recorded_at TIMESTAMPTZ DEFAULT NOW() +); + +-- Index for pattern lookup +CREATE INDEX IF NOT EXISTS idx_patterns_table +ON ruvector_dag.patterns (table_name); + +-- Index for trajectory analysis +CREATE INDEX IF NOT EXISTS idx_trajectories_table_time +ON ruvector_dag.trajectory_history (table_name, recorded_at DESC); + +-- Cleanup old trajectories (run periodically) +CREATE OR REPLACE FUNCTION ruvector_dag.cleanup_old_trajectories( + retention_days INT DEFAULT 7 +) RETURNS INT AS $$ +DECLARE + deleted_count INT; +BEGIN + DELETE FROM ruvector_dag.trajectory_history + WHERE recorded_at < NOW() - (retention_days || ' days')::INTERVAL; + + GET DIAGNOSTICS deleted_count = ROW_COUNT; + RETURN deleted_count; +END; +$$ LANGUAGE plpgsql; +``` diff --git a/docs/dag/05-QUERY-PLAN-DAG.md b/docs/dag/05-QUERY-PLAN-DAG.md new file mode 100644 index 000000000..9134214e4 --- /dev/null +++ b/docs/dag/05-QUERY-PLAN-DAG.md @@ -0,0 +1,914 @@ +# Query Plan as Learnable DAG + +## Overview + +This document specifies how PostgreSQL query plans are represented as DAGs (Directed Acyclic Graphs) and how they become targets for neural learning. + +## Query Plan DAG Structure + +### Conceptual Model + +``` + ┌─────────────┐ + │ RESULT │ (Root) + └──────┬──────┘ + │ + ┌──────┴──────┐ + │ SORT │ + └──────┬──────┘ + │ + ┌────────────┴────────────┐ + │ │ + ┌──────┴──────┐ ┌──────┴──────┐ + │ FILTER │ │ FILTER │ + └──────┬──────┘ └──────┬──────┘ + │ │ + ┌──────┴──────┐ ┌──────┴──────┐ + │ HNSW SCAN │ │ SEQ SCAN │ + │ (documents) │ │ (authors) │ + └─────────────┘ └─────────────┘ + + Leaf Nodes Leaf Nodes +``` + +### NeuralDagPlan Structure + +```rust +/// Query plan enhanced with neural learning capabilities +#[derive(Clone, Debug)] +pub struct NeuralDagPlan { + // ═══════════════════════════════════════════════════════════════ + // BASE PLAN STRUCTURE + // ═══════════════════════════════════════════════════════════════ + + /// Plan ID (unique per execution) + pub plan_id: u64, + + /// Root operator + pub root: OperatorNode, + + /// All operators in topological order (leaves first) + pub operators: Vec, + + /// Edges: parent_id -> Vec + pub edges: HashMap>, + + /// Reverse edges: child_id -> parent_id + pub reverse_edges: HashMap, + + /// Pipeline breakers (blocking operators) + pub pipeline_breakers: Vec, + + /// Parallelism configuration + pub parallelism: usize, + + // ═══════════════════════════════════════════════════════════════ + // NEURAL ENHANCEMENTS + // ═══════════════════════════════════════════════════════════════ + + /// Operator embeddings (256-dim per operator) + pub operator_embeddings: Vec>, + + /// Plan embedding (computed from operators) + pub plan_embedding: Option>, + + /// Attention weights between operators + pub attention_weights: Vec>, + + /// Selected attention type + pub attention_type: DagAttentionType, + + /// Trajectory ID (links to ReasoningBank) + pub trajectory_id: Option, + + // ═══════════════════════════════════════════════════════════════ + // LEARNED PARAMETERS + // ═══════════════════════════════════════════════════════════════ + + /// Learned cost estimates per operator + pub learned_costs: Option>, + + /// Execution parameters + pub params: ExecutionParams, + + /// Pattern match info (if pattern was applied) + pub pattern_match: Option, + + // ═══════════════════════════════════════════════════════════════ + // OPTIMIZATION STATE + // ═══════════════════════════════════════════════════════════════ + + /// MinCut criticality per operator + pub criticalities: Option>, + + /// Critical path operators + pub critical_path: Option>, + + /// Bottleneck score (0.0 - 1.0) + pub bottleneck_score: Option, +} + +/// Single operator in the plan DAG +#[derive(Clone, Debug)] +pub struct OperatorNode { + /// Unique operator ID + pub id: OperatorId, + + /// Operator type + pub op_type: OperatorType, + + /// Target table (if applicable) + pub table_name: Option, + + /// Index used (if applicable) + pub index_name: Option, + + /// Filter predicate (if applicable) + pub filter: Option, + + /// Join condition (if join) + pub join_condition: Option, + + /// Projected columns + pub projection: Vec, + + /// Estimated rows + pub estimated_rows: f64, + + /// Estimated cost + pub estimated_cost: f64, + + /// Operator embedding (learned) + pub embedding: Vec, + + /// Depth in DAG (0 = leaf) + pub depth: usize, + + /// Is this on critical path? + pub is_critical: bool, + + /// MinCut criticality score + pub criticality: f32, +} + +/// Operator types +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub enum OperatorType { + // Scan operators (leaves) + SeqScan, + IndexScan, + IndexOnlyScan, + HnswScan, + IvfFlatScan, + BitmapScan, + + // Join operators + NestedLoop, + HashJoin, + MergeJoin, + + // Aggregation operators + Aggregate, + GroupAggregate, + HashAggregate, + + // Sort operators + Sort, + IncrementalSort, + + // Filter operators + Filter, + Result, + + // Set operators + Append, + MergeAppend, + Union, + Intersect, + Except, + + // Subquery operators + SubqueryScan, + CteScan, + MaterializeNode, + + // Utility + Limit, + Unique, + WindowAgg, + + // Parallel + Gather, + GatherMerge, +} + +/// Pattern match information +#[derive(Clone, Debug)] +pub struct PatternMatch { + pub pattern_id: PatternId, + pub confidence: f32, + pub similarity: f32, + pub applied_params: ExecutionParams, +} +``` + +### Operator Embedding + +```rust +impl OperatorNode { + /// Generate embedding for this operator + pub fn generate_embedding(&mut self, config: &EmbeddingConfig) { + let dim = config.hidden_dim; + let mut embedding = vec![0.0; dim]; + + // 1. Operator type encoding (one-hot style, but dense) + let type_offset = self.op_type.type_index() * 16; + for i in 0..16 { + embedding[type_offset + i] = self.op_type.type_features()[i]; + } + + // 2. Cardinality encoding (log scale) + let card_offset = 128; + let log_rows = (self.estimated_rows + 1.0).ln(); + embedding[card_offset] = log_rows / 20.0; // Normalize + + // 3. Cost encoding (log scale) + let cost_offset = 129; + let log_cost = (self.estimated_cost + 1.0).ln(); + embedding[cost_offset] = log_cost / 30.0; // Normalize + + // 4. Depth encoding + let depth_offset = 130; + embedding[depth_offset] = self.depth as f32 / 20.0; + + // 5. Table/index encoding (if applicable) + if let Some(ref table) = self.table_name { + let table_hash = hash_string(table); + let table_offset = 132; + for i in 0..16 { + embedding[table_offset + i] = ((table_hash >> (i * 4)) & 0xF) as f32 / 16.0; + } + } + + // 6. Filter complexity encoding + if let Some(ref filter) = self.filter { + let filter_offset = 148; + embedding[filter_offset] = filter.complexity() as f32 / 10.0; + embedding[filter_offset + 1] = filter.selectivity_estimate(); + } + + // 7. Join encoding + if let Some(ref join) = self.join_condition { + let join_offset = 150; + embedding[join_offset] = join.join_type.type_index() as f32 / 4.0; + embedding[join_offset + 1] = join.estimated_selectivity; + } + + // L2 normalize + let norm: f32 = embedding.iter().map(|x| x * x).sum::().sqrt(); + if norm > 1e-8 { + for x in &mut embedding { + *x /= norm; + } + } + + self.embedding = embedding; + } +} + +impl OperatorType { + /// Get feature vector for operator type + fn type_features(&self) -> [f32; 16] { + match self { + // Scans - low cost per row + OperatorType::SeqScan => [1.0, 0.0, 0.0, 0.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], + OperatorType::IndexScan => [0.8, 0.2, 0.0, 0.0, 0.1, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], + OperatorType::HnswScan => [0.6, 0.4, 0.0, 0.0, 0.05, 0.8, 0.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], + OperatorType::IvfFlatScan => [0.7, 0.3, 0.0, 0.0, 0.08, 0.7, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], + + // Joins - high cost + OperatorType::NestedLoop => [0.0, 0.0, 1.0, 0.0, 0.9, 0.0, 0.0, 0.8, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], + OperatorType::HashJoin => [0.0, 0.0, 0.8, 0.2, 0.5, 0.0, 0.0, 0.6, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], + OperatorType::MergeJoin => [0.0, 0.0, 0.6, 0.4, 0.4, 0.0, 0.0, 0.4, 0.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], + + // Aggregation - blocking + OperatorType::Aggregate => [0.0, 0.0, 0.0, 0.0, 0.3, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.5, 0.0, 0.0, 0.0, 0.0], + OperatorType::HashAggregate => [0.0, 0.0, 0.0, 0.0, 0.4, 0.0, 0.0, 0.0, 0.5, 0.8, 0.0, 0.6, 0.0, 0.0, 0.0, 0.0], + + // Sort - blocking + OperatorType::Sort => [0.0, 0.0, 0.0, 0.0, 0.6, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.7, 0.0, 0.0, 0.0, 0.0], + + // Default + _ => [0.5; 16], + } + } + + fn type_index(&self) -> usize { + match self { + OperatorType::SeqScan => 0, + OperatorType::IndexScan => 1, + OperatorType::IndexOnlyScan => 1, + OperatorType::HnswScan => 2, + OperatorType::IvfFlatScan => 3, + OperatorType::BitmapScan => 4, + OperatorType::NestedLoop => 5, + OperatorType::HashJoin => 6, + OperatorType::MergeJoin => 7, + OperatorType::Aggregate | OperatorType::GroupAggregate | OperatorType::HashAggregate => 8, + OperatorType::Sort | OperatorType::IncrementalSort => 9, + OperatorType::Filter => 10, + OperatorType::Limit => 11, + _ => 12, + } + } +} +``` + +## Plan Conversion from PostgreSQL + +### PlannedStmt to NeuralDagPlan + +```rust +impl NeuralDagPlan { + /// Convert PostgreSQL PlannedStmt to NeuralDagPlan + pub unsafe fn from_planned_stmt(stmt: *mut pg_sys::PlannedStmt) -> Self { + let mut plan = NeuralDagPlan::new(); + + // Extract plan tree + let plan_tree = (*stmt).planTree; + plan.root = Self::convert_plan_node(plan_tree, &mut plan, 0); + + // Compute topological order + plan.compute_topological_order(); + + // Generate embeddings + plan.generate_embeddings(); + + // Identify pipeline breakers + plan.identify_pipeline_breakers(); + + plan + } + + /// Recursively convert plan nodes + unsafe fn convert_plan_node( + node: *mut pg_sys::Plan, + plan: &mut NeuralDagPlan, + depth: usize, + ) -> OperatorNode { + if node.is_null() { + panic!("Null plan node"); + } + + let node_type = (*node).type_; + let estimated_rows = (*node).plan_rows; + let estimated_cost = (*node).total_cost; + + let op_type = Self::pg_node_to_op_type(node_type, node); + let op_id = plan.next_operator_id(); + + let mut operator = OperatorNode { + id: op_id, + op_type, + table_name: Self::extract_table_name(node), + index_name: Self::extract_index_name(node), + filter: Self::extract_filter(node), + join_condition: Self::extract_join_condition(node), + projection: Self::extract_projection(node), + estimated_rows, + estimated_cost, + embedding: vec![], + depth, + is_critical: false, + criticality: 0.0, + }; + + // Process children + let left_plan = (*node).lefttree; + let right_plan = (*node).righttree; + + let mut child_ids = Vec::new(); + + if !left_plan.is_null() { + let left_op = Self::convert_plan_node(left_plan, plan, depth + 1); + child_ids.push(left_op.id); + plan.reverse_edges.insert(left_op.id, op_id); + plan.operators.push(left_op); + } + + if !right_plan.is_null() { + let right_op = Self::convert_plan_node(right_plan, plan, depth + 1); + child_ids.push(right_op.id); + plan.reverse_edges.insert(right_op.id, op_id); + plan.operators.push(right_op); + } + + if !child_ids.is_empty() { + plan.edges.insert(op_id, child_ids); + } + + operator + } + + /// Map PostgreSQL node type to OperatorType + unsafe fn pg_node_to_op_type(node_type: pg_sys::NodeTag, node: *mut pg_sys::Plan) -> OperatorType { + match node_type { + pg_sys::NodeTag::T_SeqScan => OperatorType::SeqScan, + pg_sys::NodeTag::T_IndexScan => { + // Check if it's HNSW or IVFFlat + let index_scan = node as *mut pg_sys::IndexScan; + let index_oid = (*index_scan).indexid; + + if Self::is_hnsw_index(index_oid) { + OperatorType::HnswScan + } else if Self::is_ivfflat_index(index_oid) { + OperatorType::IvfFlatScan + } else { + OperatorType::IndexScan + } + } + pg_sys::NodeTag::T_IndexOnlyScan => OperatorType::IndexOnlyScan, + pg_sys::NodeTag::T_BitmapHeapScan => OperatorType::BitmapScan, + pg_sys::NodeTag::T_NestLoop => OperatorType::NestedLoop, + pg_sys::NodeTag::T_HashJoin => OperatorType::HashJoin, + pg_sys::NodeTag::T_MergeJoin => OperatorType::MergeJoin, + pg_sys::NodeTag::T_Agg => { + let agg = node as *mut pg_sys::Agg; + match (*agg).aggstrategy { + pg_sys::AggStrategy::AGG_HASHED => OperatorType::HashAggregate, + pg_sys::AggStrategy::AGG_SORTED => OperatorType::GroupAggregate, + _ => OperatorType::Aggregate, + } + } + pg_sys::NodeTag::T_Sort => OperatorType::Sort, + pg_sys::NodeTag::T_IncrementalSort => OperatorType::IncrementalSort, + pg_sys::NodeTag::T_Limit => OperatorType::Limit, + pg_sys::NodeTag::T_Unique => OperatorType::Unique, + pg_sys::NodeTag::T_Append => OperatorType::Append, + pg_sys::NodeTag::T_MergeAppend => OperatorType::MergeAppend, + pg_sys::NodeTag::T_Gather => OperatorType::Gather, + pg_sys::NodeTag::T_GatherMerge => OperatorType::GatherMerge, + pg_sys::NodeTag::T_WindowAgg => OperatorType::WindowAgg, + pg_sys::NodeTag::T_SubqueryScan => OperatorType::SubqueryScan, + pg_sys::NodeTag::T_CteScan => OperatorType::CteScan, + pg_sys::NodeTag::T_Material => OperatorType::MaterializeNode, + pg_sys::NodeTag::T_Result => OperatorType::Result, + _ => OperatorType::Filter, // Default + } + } +} +``` + +## Plan Embedding Computation + +### Hierarchical Aggregation + +```rust +impl NeuralDagPlan { + /// Generate plan-level embedding from operator embeddings + pub fn generate_plan_embedding(&mut self) { + let dim = self.operator_embeddings[0].len(); + let mut plan_embedding = vec![0.0; dim]; + + // Method 1: Weighted sum by depth (deeper = lower weight) + let max_depth = self.operators.iter().map(|o| o.depth).max().unwrap_or(0); + + for (i, op) in self.operators.iter().enumerate() { + let depth_weight = 1.0 / (op.depth as f32 + 1.0); + let cost_weight = (op.estimated_cost / self.total_cost()).min(1.0) as f32; + let weight = depth_weight * 0.5 + cost_weight * 0.5; + + for (j, &val) in self.operator_embeddings[i].iter().enumerate() { + plan_embedding[j] += weight * val; + } + } + + // L2 normalize + let norm: f32 = plan_embedding.iter().map(|x| x * x).sum::().sqrt(); + if norm > 1e-8 { + for x in &mut plan_embedding { + *x /= norm; + } + } + + self.plan_embedding = Some(plan_embedding); + } + + /// Generate embedding using attention over operators + pub fn generate_plan_embedding_with_attention(&mut self, attention: &dyn DagAttention) { + // Use root operator as query + let root_embedding = &self.operator_embeddings[0]; + + // Build context from all operators + let ctx = self.build_dag_context(); + + // Compute attention weights + let query_node = DagNode { + id: self.root.id, + embedding: root_embedding.clone(), + }; + + let output = attention.forward(&query_node, &ctx, &AttentionConfig::default()) + .expect("Attention computation failed"); + + // Store attention weights + self.attention_weights = vec![output.weights.clone()]; + + // Use aggregated output as plan embedding + self.plan_embedding = Some(output.aggregated); + } + + fn build_dag_context(&self) -> DagContext { + DagContext { + nodes: self.operators.iter() + .map(|op| DagNode { + id: op.id, + embedding: op.embedding.clone(), + }) + .collect(), + edges: self.edges.clone(), + reverse_edges: self.reverse_edges.iter() + .map(|(&child, &parent)| (child, vec![parent])) + .collect(), + depths: self.operators.iter() + .map(|op| (op.id, op.depth)) + .collect(), + timestamps: None, + criticalities: self.criticalities.as_ref().map(|c| { + self.operators.iter() + .enumerate() + .map(|(i, op)| (op.id, c[i])) + .collect() + }), + } + } +} +``` + +## Plan Optimization + +### Learned Cost Adjustment + +```rust +impl NeuralDagPlan { + /// Apply learned cost adjustments + pub fn apply_learned_costs(&mut self) { + if let Some(ref learned_costs) = self.learned_costs { + for (i, op) in self.operators.iter_mut().enumerate() { + if i < learned_costs.len() { + // Adjust estimated cost by learned factor + let adjustment = learned_costs[i]; + op.estimated_cost *= (1.0 + adjustment) as f64; + } + } + } + } + + /// Reorder operators based on learned pattern + pub fn reorder_operators(&mut self, optimal_ordering: &[OperatorId]) { + // Only reorder within commutative operators (e.g., join order) + let join_ops: Vec<_> = self.operators.iter() + .filter(|op| matches!(op.op_type, + OperatorType::HashJoin | + OperatorType::MergeJoin | + OperatorType::NestedLoop)) + .map(|op| op.id) + .collect(); + + if join_ops.len() < 2 { + return; // Nothing to reorder + } + + // Apply learned ordering + // This is a simplified version - real implementation needs + // to preserve DAG constraints + for (i, &target_id) in optimal_ordering.iter().enumerate() { + if i < join_ops.len() { + // Swap join operators to match target ordering + // (preserving child relationships) + } + } + } + + /// Apply learned execution parameters + pub fn apply_params(&mut self, params: &ExecutionParams) { + self.params = params.clone(); + + // Apply to relevant operators + for op in &mut self.operators { + match op.op_type { + OperatorType::HnswScan => { + if let Some(ef) = params.ef_search { + op.embedding[160] = ef as f32 / 100.0; // Encode in embedding + } + } + OperatorType::IvfFlatScan => { + if let Some(probes) = params.probes { + op.embedding[161] = probes as f32 / 50.0; + } + } + _ => {} + } + } + } +} +``` + +### Critical Path Analysis + +```rust +impl NeuralDagPlan { + /// Compute critical path through the plan DAG + pub fn compute_critical_path(&mut self) { + // Dynamic programming: longest path + let mut longest_to: HashMap = HashMap::new(); + let mut longest_from: HashMap = HashMap::new(); + let mut predecessor: HashMap = HashMap::new(); + + // Forward pass (leaves to root) - longest path TO each node + for op in self.operators.iter().rev() { // Reverse topo order + let mut max_cost = 0.0; + let mut max_pred = None; + + if let Some(children) = self.edges.get(&op.id) { + for &child_id in children { + let child_cost = longest_to.get(&child_id).unwrap_or(&0.0); + if *child_cost > max_cost { + max_cost = *child_cost; + max_pred = Some(child_id); + } + } + } + + longest_to.insert(op.id, max_cost + op.estimated_cost); + if let Some(pred) = max_pred { + predecessor.insert(op.id, pred); + } + } + + // Backward pass (root to leaves) - longest path FROM each node + for op in &self.operators { + let mut max_cost = 0.0; + + if let Some(&parent_id) = self.reverse_edges.get(&op.id) { + let parent_cost = longest_from.get(&parent_id).unwrap_or(&0.0); + max_cost = max_cost.max(*parent_cost + self.get_operator(parent_id).estimated_cost); + } + + longest_from.insert(op.id, max_cost); + } + + // Find critical path + let global_longest = longest_to.values().cloned().fold(0.0, f64::max); + + let mut critical_path = Vec::new(); + for op in &self.operators { + let total_through = longest_to[&op.id] + longest_from[&op.id]; + if (total_through - global_longest).abs() < 1e-6 { + critical_path.push(op.id); + } + } + + // Mark operators + for op in &mut self.operators { + op.is_critical = critical_path.contains(&op.id); + } + + self.critical_path = Some(critical_path); + } + + /// Compute bottleneck score (0.0 - 1.0) + pub fn compute_bottleneck_score(&mut self) { + if let Some(ref critical_path) = self.critical_path { + if critical_path.is_empty() { + self.bottleneck_score = Some(0.0); + return; + } + + // Bottleneck = max(single_op_cost / total_cost) + let total_cost = self.total_cost(); + let max_single = critical_path.iter() + .map(|&id| self.get_operator(id).estimated_cost) + .fold(0.0, f64::max); + + self.bottleneck_score = Some((max_single / total_cost) as f32); + } + } +} +``` + +## Learning Target: Plan Quality + +### Quality Computation + +```rust +/// Compute quality score for a plan execution +pub fn compute_plan_quality(plan: &NeuralDagPlan, metrics: &ExecutionMetrics) -> f32 { + // Multi-objective quality function + + // 1. Latency score (lower is better) + // Target: 10ms for simple queries, 1s for complex + let complexity = plan.operators.len() as f32; + let target_latency_us = 10000.0 * complexity.sqrt(); + let latency_score = (target_latency_us / (metrics.latency_us as f32 + 1.0)).min(1.0); + + // 2. Accuracy score (for vector queries) + // If we have relevance feedback + let accuracy_score = if let Some(precision) = metrics.precision { + precision + } else { + 1.0 // Assume accurate if no feedback + }; + + // 3. Efficiency score (rows per microsecond) + let efficiency_score = if metrics.latency_us > 0 { + (metrics.rows_processed as f32 / metrics.latency_us as f32 * 1000.0).min(1.0) + } else { + 1.0 + }; + + // 4. Memory score (lower is better) + let target_memory = 10_000_000.0 * complexity; // 10MB per operator + let memory_score = (target_memory / (metrics.memory_bytes as f32 + 1.0)).min(1.0); + + // 5. Cache efficiency + let cache_score = metrics.cache_hit_rate; + + // Weighted combination + let weights = [0.35, 0.25, 0.15, 0.15, 0.10]; + let scores = [latency_score, accuracy_score, efficiency_score, memory_score, cache_score]; + + weights.iter().zip(scores.iter()) + .map(|(w, s)| w * s) + .sum() +} +``` + +### Gradient Estimation + +```rust +impl DagTrajectory { + /// Estimate gradient for REINFORCE-style learning + pub fn estimate_gradient(&self) -> Vec { + let dim = self.plan_embedding.len(); + let mut gradient = vec![0.0; dim]; + + // REINFORCE with baseline + let baseline = 0.5; // Could be learned + let advantage = self.quality - baseline; + + // gradient += advantage * activation + // Simplified: use plan embedding as "activation" + for (i, &val) in self.plan_embedding.iter().enumerate() { + gradient[i] = advantage * val; + } + + // Also incorporate operator-level signals + for (op_idx, op_embedding) in self.operator_embeddings.iter().enumerate() { + // Weight by attention + let attention_weight = if op_idx < self.attention_weights.len() { + self.attention_weights.get(0) + .and_then(|w| w.get(op_idx)) + .unwrap_or(&(1.0 / self.operator_embeddings.len() as f32)) + .clone() + } else { + 1.0 / self.operator_embeddings.len() as f32 + }; + + for (i, &val) in op_embedding.iter().enumerate() { + if i < dim { + gradient[i] += advantage * val * attention_weight * 0.5; + } + } + } + + // L2 normalize + let norm: f32 = gradient.iter().map(|x| x * x).sum::().sqrt(); + if norm > 1e-8 { + for x in &mut gradient { + *x /= norm; + } + } + + gradient + } +} +``` + +## Integration with PostgreSQL Planner + +### Plan Modification Points + +```rust +/// Points where neural DAG can influence planning +pub enum PlanModificationPoint { + /// Before any planning + PrePlanning, + + /// After join enumeration, before selecting best join order + JoinOrdering, + + /// After creating base plan, before optimization + PreOptimization, + + /// After optimization, before execution + PostOptimization, + + /// During execution (adaptive) + DuringExecution, +} + +impl NeuralDagPlan { + /// Apply neural modifications at specified point + pub fn apply_modifications(&mut self, point: PlanModificationPoint, engine: &DagSonaEngine) { + match point { + PlanModificationPoint::PrePlanning => { + // Hint optimal parameters based on query pattern + self.apply_pre_planning_hints(engine); + } + + PlanModificationPoint::JoinOrdering => { + // Suggest optimal join order + if let Some(ordering) = engine.suggest_join_order(&self.plan_embedding) { + self.reorder_operators(&ordering); + } + } + + PlanModificationPoint::PreOptimization => { + // Adjust cost estimates + if let Some(costs) = engine.predict_costs(&self.plan_embedding) { + self.learned_costs = Some(costs); + self.apply_learned_costs(); + } + } + + PlanModificationPoint::PostOptimization => { + // Final parameter tuning + if let Some(params) = engine.suggest_params(&self.plan_embedding) { + self.apply_params(¶ms); + } + } + + PlanModificationPoint::DuringExecution => { + // Adaptive re-planning (future work) + } + } + } +} +``` + +## Serialization + +### Plan Persistence + +```rust +impl NeuralDagPlan { + /// Serialize plan for storage + pub fn to_bytes(&self) -> Vec { + bincode::serialize(self).expect("Serialization failed") + } + + /// Deserialize plan + pub fn from_bytes(bytes: &[u8]) -> Result { + bincode::deserialize(bytes) + } + + /// Export to JSON for debugging + pub fn to_json(&self) -> serde_json::Value { + json!({ + "plan_id": self.plan_id, + "operators": self.operators.iter().map(|op| json!({ + "id": op.id, + "type": format!("{:?}", op.op_type), + "table": op.table_name, + "estimated_rows": op.estimated_rows, + "estimated_cost": op.estimated_cost, + "depth": op.depth, + "is_critical": op.is_critical, + "criticality": op.criticality, + })).collect::>(), + "edges": self.edges, + "attention_type": format!("{:?}", self.attention_type), + "bottleneck_score": self.bottleneck_score, + "params": { + "ef_search": self.params.ef_search, + "probes": self.params.probes, + "parallelism": self.params.parallelism, + } + }) + } +} +``` + +## Performance Considerations + +| Operation | Complexity | Target Latency | +|-----------|------------|----------------| +| Plan conversion | O(n) | <1ms | +| Embedding generation | O(n × d) | <500μs | +| Plan embedding | O(n × d) | <200μs | +| Critical path | O(n²) | <1ms | +| MinCut criticality | O(n^0.12) | <10ms | +| Pattern matching | O(k × d) | <1ms | + +Where n = operators, d = embedding dimension (256), k = patterns (100). diff --git a/docs/dag/06-MINCUT-OPTIMIZATION.md b/docs/dag/06-MINCUT-OPTIMIZATION.md new file mode 100644 index 000000000..c9e1a3c4d --- /dev/null +++ b/docs/dag/06-MINCUT-OPTIMIZATION.md @@ -0,0 +1,667 @@ +# MinCut Optimization Specification + +## Overview + +This document specifies how the subpolynomial O(n^0.12) min-cut algorithm from `ruvector-mincut` integrates with the Neural DAG system for bottleneck detection and optimization. + +## MinCut Integration Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ MINCUT OPTIMIZATION LAYER │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ SUBPOLYNOMIAL MINCUT ENGINE │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ Hierarchical│ │ LocalKCut │ │ LinkCut │ │ │ +│ │ │Decomposition│ │ Oracle │ │ Tree │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ DAG CRITICALITY ANALYZER │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ Operator │ │ Bottleneck │ │ Critical │ │ │ +│ │ │ Criticality │ │ Detection │ │ Path │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ OPTIMIZATION ACTIONS │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │ +│ │ │ Gated │ │ Redundancy │ │ Parallel │ │ Self- │ │ │ +│ │ │ Attention │ │ Injection │ │ Expansion │ │ Healing │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## DAG MinCut Engine + +### Core Structure + +```rust +/// MinCut engine adapted for query plan DAGs +pub struct DagMinCutEngine { + /// Subpolynomial min-cut algorithm + mincut: SubpolynomialMinCut, + + /// Graph representation of current DAG + graph: DynamicGraph, + + /// Cached criticality scores + criticality_cache: DashMap, + + /// Configuration + config: MinCutConfig, + + /// Metrics + metrics: MinCutMetrics, +} + +#[derive(Clone, Debug)] +pub struct MinCutConfig { + /// Enable/disable mincut analysis + pub enabled: bool, + + /// Criticality threshold for bottleneck detection + pub bottleneck_threshold: f32, + + /// Maximum operators to analyze + pub max_operators: usize, + + /// Cache TTL in seconds + pub cache_ttl_secs: u64, + + /// Enable self-healing + pub self_healing_enabled: bool, + + /// Healing check interval + pub healing_interval_ms: u64, +} + +impl Default for MinCutConfig { + fn default() -> Self { + Self { + enabled: true, + bottleneck_threshold: 0.5, + max_operators: 1000, + cache_ttl_secs: 300, + self_healing_enabled: true, + healing_interval_ms: 300000, // 5 minutes + } + } +} +``` + +### Graph Construction from DAG + +```rust +impl DagMinCutEngine { + /// Build graph from query plan DAG + pub fn build_from_plan(&mut self, plan: &NeuralDagPlan) { + self.graph.clear(); + + // Add vertices (operators) + for op in &plan.operators { + let weight = self.operator_weight(op); + self.graph.add_vertex(op.id, weight); + } + + // Add edges (data flow) + for (&parent_id, children) in &plan.edges { + for &child_id in children { + // Edge weight = data volume estimate + let parent_op = plan.get_operator(parent_id); + let weight = parent_op.estimated_rows as f64; + self.graph.add_edge(parent_id, child_id, weight); + } + } + + // Initialize min-cut structure + self.mincut.initialize(&self.graph); + } + + /// Compute operator weight for min-cut + fn operator_weight(&self, op: &OperatorNode) -> f64 { + // Weight based on: + // 1. Estimated cost (primary) + // 2. Blocking nature (pipeline breakers are heavier) + // 3. Parallelizability (less parallelizable = heavier) + + let base_weight = op.estimated_cost; + + let blocking_factor = if op.is_pipeline_breaker() { + 2.0 + } else { + 1.0 + }; + + let parallel_factor = match op.op_type { + OperatorType::Sort | OperatorType::Aggregate => 1.5, + OperatorType::HashJoin => 1.2, + _ => 1.0, + }; + + base_weight * blocking_factor * parallel_factor + } +} +``` + +## Criticality Computation + +### Operator Criticality + +```rust +impl DagMinCutEngine { + /// Compute criticality for all operators + pub fn compute_all_criticalities(&self, plan: &NeuralDagPlan) -> HashMap { + let global_cut = self.mincut.query(); + let mut criticalities = HashMap::new(); + + for op in &plan.operators { + let criticality = self.compute_operator_criticality(op.id, global_cut); + criticalities.insert(op.id, criticality); + } + + criticalities + } + + /// Compute criticality for a single operator + /// Criticality = how much removing this operator would reduce min-cut + pub fn compute_operator_criticality(&self, op_id: OperatorId, global_cut: u64) -> f32 { + // Check cache first + if let Some(cached) = self.criticality_cache.get(&op_id) { + return *cached; + } + + // Use LocalKCut oracle + let query = LocalKCutQuery { + seed_vertices: vec![op_id], + budget_k: global_cut, + radius: 3, // Local neighborhood + }; + + let criticality = match self.mincut.local_query(query) { + LocalKCutResult::Found { cut_value, .. } => { + // Criticality = (global - local) / global + if global_cut > 0 { + (global_cut - cut_value) as f32 / global_cut as f32 + } else { + 0.0 + } + } + LocalKCutResult::NoneInLocality => 0.0, + }; + + // Cache result + self.criticality_cache.insert(op_id, criticality); + + criticality + } + + /// Identify bottleneck operators + pub fn identify_bottlenecks(&self, plan: &NeuralDagPlan) -> Vec { + let criticalities = self.compute_all_criticalities(plan); + + let mut bottlenecks: Vec<_> = criticalities.iter() + .filter(|(_, &crit)| crit > self.config.bottleneck_threshold) + .map(|(&op_id, &crit)| { + let op = plan.get_operator(op_id); + BottleneckInfo { + operator_id: op_id, + operator_type: op.op_type.clone(), + criticality: crit, + estimated_cost: op.estimated_cost, + recommendation: self.generate_recommendation(op, crit), + } + }) + .collect(); + + // Sort by criticality (most critical first) + bottlenecks.sort_by(|a, b| b.criticality.partial_cmp(&a.criticality).unwrap()); + + bottlenecks + } + + /// Generate optimization recommendation for bottleneck + fn generate_recommendation(&self, op: &OperatorNode, criticality: f32) -> OptimizationRecommendation { + match op.op_type { + OperatorType::SeqScan => { + OptimizationRecommendation::CreateIndex { + table: op.table_name.clone().unwrap_or_default(), + columns: op.filter.as_ref() + .map(|f| f.columns()) + .unwrap_or_default(), + } + } + + OperatorType::HnswScan | OperatorType::IvfFlatScan => { + if criticality > 0.8 { + OptimizationRecommendation::IncreaseEfSearch { + current: 40, // Would be extracted from plan + recommended: 80, + } + } else { + OptimizationRecommendation::None + } + } + + OperatorType::NestedLoop => { + OptimizationRecommendation::ConsiderHashJoin { + estimated_improvement: criticality * 50.0, + } + } + + OperatorType::Sort => { + if op.estimated_rows > 100000.0 { + OptimizationRecommendation::AddSortIndex { + columns: op.projection.clone(), + } + } else { + OptimizationRecommendation::None + } + } + + OperatorType::HashAggregate if op.estimated_rows > 1000000.0 => { + OptimizationRecommendation::ConsiderPartitioning { + partition_key: op.projection.first().cloned(), + } + } + + _ => OptimizationRecommendation::None, + } + } +} + +/// Information about a bottleneck +#[derive(Clone, Debug)] +pub struct BottleneckInfo { + pub operator_id: OperatorId, + pub operator_type: OperatorType, + pub criticality: f32, + pub estimated_cost: f64, + pub recommendation: OptimizationRecommendation, +} + +/// Optimization recommendations +#[derive(Clone, Debug)] +pub enum OptimizationRecommendation { + None, + CreateIndex { table: String, columns: Vec }, + IncreaseEfSearch { current: usize, recommended: usize }, + ConsiderHashJoin { estimated_improvement: f32 }, + AddSortIndex { columns: Vec }, + ConsiderPartitioning { partition_key: Option }, + AddParallelism { recommended_workers: usize }, + MaterializeSubquery { subquery_id: OperatorId }, +} +``` + +## MinCut Gated Attention Integration + +### Gating Mechanism + +```rust +impl DagMinCutEngine { + /// Compute attention gates based on criticality + pub fn compute_attention_gates( + &self, + plan: &NeuralDagPlan, + ) -> Vec { + let criticalities = self.compute_all_criticalities(plan); + + plan.operators.iter() + .map(|op| { + let crit = criticalities.get(&op.id).unwrap_or(&0.0); + + if *crit > self.config.bottleneck_threshold { + 1.0 // Full attention for bottlenecks + } else { + crit / self.config.bottleneck_threshold // Scaled + } + }) + .collect() + } + + /// Apply gating to attention weights + pub fn gate_attention_weights( + &self, + weights: &[f32], + gates: &[f32], + ) -> Vec { + assert_eq!(weights.len(), gates.len()); + + let gated: Vec = weights.iter() + .zip(gates.iter()) + .map(|(w, g)| w * g) + .collect(); + + // Renormalize + let sum: f32 = gated.iter().sum(); + if sum > 1e-8 { + gated.iter().map(|w| w / sum).collect() + } else { + vec![1.0 / weights.len() as f32; weights.len()] + } + } +} +``` + +## Dynamic Updates + +### Incremental MinCut Maintenance + +```rust +impl DagMinCutEngine { + /// Handle operator cost update (O(n^0.12) amortized) + pub fn update_operator_cost(&mut self, op_id: OperatorId, new_cost: f64) { + let old_weight = self.graph.get_vertex_weight(op_id); + let new_weight = new_cost * self.get_operator_factors(op_id); + + // Update graph + self.graph.update_vertex_weight(op_id, new_weight); + + // Incremental min-cut update + // The subpolynomial algorithm handles this efficiently + self.mincut.on_vertex_weight_change(op_id, old_weight, new_weight); + + // Invalidate cache for affected operators + self.invalidate_local_cache(op_id); + } + + /// Handle edge addition (e.g., plan change) + pub fn add_edge(&mut self, from: OperatorId, to: OperatorId, weight: f64) { + self.graph.add_edge(from, to, weight); + self.mincut.insert_edge(from, to); + self.invalidate_local_cache(from); + self.invalidate_local_cache(to); + } + + /// Handle edge removal + pub fn remove_edge(&mut self, from: OperatorId, to: OperatorId) { + self.graph.remove_edge(from, to); + self.mincut.delete_edge(from, to); + self.invalidate_local_cache(from); + self.invalidate_local_cache(to); + } + + /// Invalidate cache for operator and neighbors + fn invalidate_local_cache(&self, op_id: OperatorId) { + self.criticality_cache.remove(&op_id); + + // Also invalidate neighbors (within radius 3) + let neighbors = self.graph.get_neighbors_within_radius(op_id, 3); + for neighbor in neighbors { + self.criticality_cache.remove(&neighbor); + } + } +} +``` + +## Self-Healing Integration + +### Bottleneck Detection Loop + +```rust +impl DagMinCutEngine { + /// Background bottleneck detection + pub fn run_health_check(&self, plan: &NeuralDagPlan) -> HealthCheckResult { + let start = Instant::now(); + + // Compute global min-cut + let global_cut = self.mincut.query(); + + // Identify bottlenecks + let bottlenecks = self.identify_bottlenecks(plan); + + // Compute health score + let health_score = self.compute_health_score(&bottlenecks); + + // Generate alerts if needed + let alerts = self.generate_alerts(&bottlenecks); + + HealthCheckResult { + global_mincut: global_cut, + health_score, + bottleneck_count: bottlenecks.len(), + severe_bottlenecks: bottlenecks.iter() + .filter(|b| b.criticality > 0.8) + .count(), + bottlenecks, + alerts, + duration: start.elapsed(), + } + } + + fn compute_health_score(&self, bottlenecks: &[BottleneckInfo]) -> f32 { + if bottlenecks.is_empty() { + return 1.0; + } + + // Score decreases with bottleneck severity + let max_criticality = bottlenecks.iter() + .map(|b| b.criticality) + .fold(0.0, f32::max); + + let avg_criticality = bottlenecks.iter() + .map(|b| b.criticality) + .sum::() / bottlenecks.len() as f32; + + 1.0 - (max_criticality * 0.6 + avg_criticality * 0.4) + } + + fn generate_alerts(&self, bottlenecks: &[BottleneckInfo]) -> Vec { + bottlenecks.iter() + .filter(|b| b.criticality > 0.7) + .map(|b| Alert { + severity: if b.criticality > 0.9 { + AlertSeverity::Critical + } else if b.criticality > 0.8 { + AlertSeverity::Warning + } else { + AlertSeverity::Info + }, + message: format!( + "Bottleneck detected: {:?} (criticality: {:.2})", + b.operator_type, b.criticality + ), + recommendation: b.recommendation.clone(), + }) + .collect() + } +} + +#[derive(Clone, Debug)] +pub struct HealthCheckResult { + pub global_mincut: u64, + pub health_score: f32, + pub bottleneck_count: usize, + pub severe_bottlenecks: usize, + pub bottlenecks: Vec, + pub alerts: Vec, + pub duration: Duration, +} + +#[derive(Clone, Debug)] +pub struct Alert { + pub severity: AlertSeverity, + pub message: String, + pub recommendation: OptimizationRecommendation, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum AlertSeverity { + Info, + Warning, + Critical, +} +``` + +## Redundancy Injection + +### Bypass Path Creation + +```rust +impl DagMinCutEngine { + /// Suggest redundant paths to reduce bottleneck impact + pub fn suggest_redundancy(&self, plan: &NeuralDagPlan) -> Vec { + let bottlenecks = self.identify_bottlenecks(plan); + let mut suggestions = Vec::new(); + + for bottleneck in &bottlenecks { + if bottleneck.criticality > 0.7 { + // Find alternative paths around this operator + let alternatives = self.find_alternative_paths( + plan, + bottleneck.operator_id, + ); + + if let Some(alt) = alternatives.first() { + suggestions.push(RedundancySuggestion { + bottleneck_id: bottleneck.operator_id, + alternative_path: alt.clone(), + estimated_improvement: self.estimate_improvement( + bottleneck, + alt, + ), + }); + } + } + } + + suggestions + } + + fn find_alternative_paths( + &self, + plan: &NeuralDagPlan, + bottleneck_id: OperatorId, + ) -> Vec { + let mut alternatives = Vec::new(); + + let bottleneck = plan.get_operator(bottleneck_id); + + match bottleneck.op_type { + OperatorType::SeqScan => { + // Alternative: Index scan if index exists + if let Some(ref table) = bottleneck.table_name { + if let Some(index) = self.find_usable_index(table, &bottleneck.filter) { + alternatives.push(AlternativePath::UseIndex { + index_name: index, + estimated_speedup: 10.0, + }); + } + } + } + + OperatorType::NestedLoop => { + // Alternative: Hash join + alternatives.push(AlternativePath::ReplaceJoin { + new_join_type: OperatorType::HashJoin, + estimated_speedup: 5.0, + }); + } + + OperatorType::Sort => { + // Alternative: Pre-sorted input via index + alternatives.push(AlternativePath::SortedIndex { + columns: bottleneck.projection.clone(), + estimated_speedup: 3.0, + }); + } + + _ => {} + } + + alternatives + } + + fn estimate_improvement( + &self, + bottleneck: &BottleneckInfo, + alternative: &AlternativePath, + ) -> f32 { + let base_cost = bottleneck.estimated_cost; + let speedup = alternative.estimated_speedup(); + + let new_cost = base_cost / speedup; + let improvement = (base_cost - new_cost) / base_cost; + + improvement as f32 * bottleneck.criticality + } +} + +#[derive(Clone, Debug)] +pub struct RedundancySuggestion { + pub bottleneck_id: OperatorId, + pub alternative_path: AlternativePath, + pub estimated_improvement: f32, +} + +#[derive(Clone, Debug)] +pub enum AlternativePath { + UseIndex { index_name: String, estimated_speedup: f64 }, + ReplaceJoin { new_join_type: OperatorType, estimated_speedup: f64 }, + SortedIndex { columns: Vec, estimated_speedup: f64 }, + Materialize { subquery_id: OperatorId, estimated_speedup: f64 }, + Parallelize { workers: usize, estimated_speedup: f64 }, +} + +impl AlternativePath { + fn estimated_speedup(&self) -> f64 { + match self { + Self::UseIndex { estimated_speedup, .. } => *estimated_speedup, + Self::ReplaceJoin { estimated_speedup, .. } => *estimated_speedup, + Self::SortedIndex { estimated_speedup, .. } => *estimated_speedup, + Self::Materialize { estimated_speedup, .. } => *estimated_speedup, + Self::Parallelize { estimated_speedup, .. } => *estimated_speedup, + } + } +} +``` + +## SQL Interface + +```sql +-- Compute mincut criticality for a plan +SELECT * FROM ruvector_dag_mincut_criticality('documents'); + +-- Get bottleneck analysis +SELECT * FROM ruvector_dag_bottlenecks('documents'); + +-- Get health check result +SELECT ruvector_dag_mincut_health('documents'); + +-- Get redundancy suggestions +SELECT * FROM ruvector_dag_redundancy_suggestions('documents'); + +-- Enable/disable mincut analysis +SET ruvector.dag_mincut_enabled = true; +SET ruvector.dag_mincut_threshold = 0.5; +``` + +## Performance Characteristics + +| Operation | Complexity | Typical Latency | +|-----------|------------|-----------------| +| Global min-cut query | O(1) | <1μs | +| Single criticality | O(n^0.12) | <100μs | +| All criticalities | O(n^1.12) | <10ms (100 ops) | +| Edge insert | O(n^0.12) amortized | <100μs | +| Edge delete | O(n^0.12) amortized | <100μs | +| Health check | O(n^1.12) | <50ms | + +## Memory Usage + +| Component | Size | Notes | +|-----------|------|-------| +| Graph structure | O(n + m) | Vertices + edges | +| Hierarchical decomposition | O(n log n) | Multi-level | +| LinkCut tree | O(n) | Sleator-Tarjan | +| Criticality cache | O(n) | Bounded by TTL | +| LocalKCut coloring | O(k² log n) | Per query | + +**Typical overhead:** ~1MB per 1000 operators diff --git a/docs/dag/07-SELF-HEALING.md b/docs/dag/07-SELF-HEALING.md new file mode 100644 index 000000000..13ce3c2c6 --- /dev/null +++ b/docs/dag/07-SELF-HEALING.md @@ -0,0 +1,1033 @@ +# Self-Healing System Specification + +## Overview + +The self-healing system automatically detects, diagnoses, and repairs issues in the Neural DAG system, including index degradation, learning drift, and performance bottlenecks. + +## Self-Healing Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ SELF-HEALING ENGINE │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────┐ │ +│ │ DETECTION LAYER │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │ +│ │ │ Anomaly │ │ Performance │ │ Index │ │ Learning │ │ │ +│ │ │ Detector │ │ Monitor │ │ Health │ │ Drift │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ DIAGNOSIS LAYER │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ Root │ │ Impact │ │ Priority │ │ │ +│ │ │ Cause │ │ Analysis │ │ Scoring │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ REPAIR LAYER │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │ +│ │ │ Index │ │ Pattern │ │ Parameter │ │ Topology │ │ │ +│ │ │ Rebalance │ │ Reset │ │ Tuning │ │ Repair │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────────┴───────────────────────────────────┐ │ +│ │ VERIFICATION LAYER │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ Repair │ │ Rollback │ │ Metrics │ │ │ +│ │ │ Validation │ │ Mechanism │ │ Reporting │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ +│ └─────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## Self-Healing Engine + +### Core Structure + +```rust +pub struct SelfHealingEngine { + /// Detection components + anomaly_detector: AnomalyDetector, + performance_monitor: PerformanceMonitor, + index_health_checker: IndexHealthChecker, + learning_drift_detector: LearningDriftDetector, + + /// Diagnosis components + root_cause_analyzer: RootCauseAnalyzer, + + /// Repair strategies + repair_strategies: Vec>, + + /// Configuration + config: HealingConfig, + + /// State tracking + active_repairs: DashMap, + repair_history: Vec, + + /// Metrics + metrics: HealingMetrics, +} + +#[derive(Clone, Debug)] +pub struct HealingConfig { + /// Enable automatic healing + pub auto_heal_enabled: bool, + + /// Check interval (milliseconds) + pub check_interval_ms: u64, + + /// Anomaly detection sensitivity (0.0 - 1.0) + pub anomaly_sensitivity: f32, + + /// Performance degradation threshold + pub performance_threshold: f32, + + /// Maximum concurrent repairs + pub max_concurrent_repairs: usize, + + /// Repair timeout (seconds) + pub repair_timeout_secs: u64, + + /// Enable rollback on failure + pub rollback_enabled: bool, + + /// History retention (days) + pub history_retention_days: u32, +} + +impl Default for HealingConfig { + fn default() -> Self { + Self { + auto_heal_enabled: true, + check_interval_ms: 300000, // 5 minutes + anomaly_sensitivity: 0.7, + performance_threshold: 0.3, // 30% degradation + max_concurrent_repairs: 3, + repair_timeout_secs: 300, + rollback_enabled: true, + history_retention_days: 30, + } + } +} +``` + +## Detection Layer + +### Anomaly Detection + +```rust +pub struct AnomalyDetector { + /// Baseline statistics + baseline: BaselineStats, + + /// Detection algorithm + algorithm: AnomalyAlgorithm, + + /// Recent observations + observations: RingBuffer, + + /// Detected anomalies + anomalies: Vec, +} + +#[derive(Clone, Debug)] +pub struct BaselineStats { + pub latency_mean: f64, + pub latency_std: f64, + pub latency_p99: f64, + + pub throughput_mean: f64, + pub throughput_std: f64, + + pub pattern_hit_rate_mean: f64, + pub pattern_hit_rate_std: f64, + + pub memory_usage_mean: f64, + pub memory_usage_std: f64, + + pub sample_count: usize, + pub last_updated: SystemTime, +} + +pub enum AnomalyAlgorithm { + /// Z-score based detection + ZScore { threshold: f32 }, + + /// Isolation Forest + IsolationForest { contamination: f32 }, + + /// Moving average deviation + MovingAverage { window: usize, threshold: f32 }, +} + +impl AnomalyDetector { + /// Check for anomalies + pub fn detect(&mut self, observation: &Observation) -> Vec { + self.observations.push(observation.clone()); + + let mut anomalies = Vec::new(); + + // Latency anomaly + if let Some(anomaly) = self.check_latency_anomaly(observation) { + anomalies.push(anomaly); + } + + // Throughput anomaly + if let Some(anomaly) = self.check_throughput_anomaly(observation) { + anomalies.push(anomaly); + } + + // Pattern hit rate anomaly + if let Some(anomaly) = self.check_hit_rate_anomaly(observation) { + anomalies.push(anomaly); + } + + // Memory anomaly + if let Some(anomaly) = self.check_memory_anomaly(observation) { + anomalies.push(anomaly); + } + + self.anomalies.extend(anomalies.clone()); + anomalies + } + + fn check_latency_anomaly(&self, obs: &Observation) -> Option { + let z_score = (obs.latency_us as f64 - self.baseline.latency_mean) + / self.baseline.latency_std; + + if z_score.abs() > 3.0 { + Some(DetectedAnomaly { + anomaly_type: AnomalyType::LatencySpike, + severity: self.compute_severity(z_score), + observed_value: obs.latency_us as f64, + expected_value: self.baseline.latency_mean, + z_score, + timestamp: obs.timestamp, + }) + } else { + None + } + } + + fn compute_severity(&self, z_score: f64) -> AnomalySeverity { + let abs_z = z_score.abs(); + if abs_z > 5.0 { + AnomalySeverity::Critical + } else if abs_z > 4.0 { + AnomalySeverity::High + } else if abs_z > 3.0 { + AnomalySeverity::Medium + } else { + AnomalySeverity::Low + } + } + + /// Update baseline with new observations + pub fn update_baseline(&mut self) { + let observations: Vec<_> = self.observations.iter().collect(); + if observations.len() < 100 { + return; // Need minimum samples + } + + // Compute new statistics + let latencies: Vec = observations.iter() + .map(|o| o.latency_us as f64) + .collect(); + + self.baseline.latency_mean = mean(&latencies); + self.baseline.latency_std = std_dev(&latencies); + self.baseline.latency_p99 = percentile(&latencies, 99.0); + + // Similar for other metrics... + + self.baseline.sample_count = observations.len(); + self.baseline.last_updated = SystemTime::now(); + } +} + +#[derive(Clone, Debug)] +pub struct DetectedAnomaly { + pub anomaly_type: AnomalyType, + pub severity: AnomalySeverity, + pub observed_value: f64, + pub expected_value: f64, + pub z_score: f64, + pub timestamp: SystemTime, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum AnomalyType { + LatencySpike, + ThroughputDrop, + HitRateDrop, + MemorySpike, + ErrorRateSpike, +} + +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum AnomalySeverity { + Low, + Medium, + High, + Critical, +} +``` + +### Index Health Checker + +```rust +pub struct IndexHealthChecker { + /// HNSW index health + hnsw_health: HashMap, + + /// IVFFlat index health + ivfflat_health: HashMap, + + /// Check history + history: Vec, +} + +#[derive(Clone, Debug)] +pub struct HnswHealth { + pub index_name: String, + pub table_name: String, + + /// Graph connectivity + pub connectivity_score: f32, + + /// Layer distribution + pub layer_distribution: Vec, + + /// Entry point quality + pub entry_point_quality: f32, + + /// Orphan nodes + pub orphan_count: usize, + + /// Fragmentation score (0.0 = none, 1.0 = severe) + pub fragmentation: f32, + + /// Last check time + pub last_checked: SystemTime, +} + +#[derive(Clone, Debug)] +pub struct IvfFlatHealth { + pub index_name: String, + pub table_name: String, + + /// Cluster balance (std dev of cluster sizes) + pub cluster_balance: f32, + + /// Empty clusters + pub empty_clusters: usize, + + /// Centroid quality + pub centroid_quality: f32, + + /// Training staleness (how old is training data) + pub training_staleness_hours: f32, + + /// Last check time + pub last_checked: SystemTime, +} + +impl IndexHealthChecker { + /// Check HNSW index health + pub fn check_hnsw(&mut self, index_name: &str) -> HnswHealth { + // Connect to PostgreSQL and analyze index + + let health = HnswHealth { + index_name: index_name.to_string(), + table_name: self.get_table_name(index_name), + connectivity_score: self.analyze_hnsw_connectivity(index_name), + layer_distribution: self.get_layer_distribution(index_name), + entry_point_quality: self.analyze_entry_point(index_name), + orphan_count: self.count_orphan_nodes(index_name), + fragmentation: self.estimate_fragmentation(index_name), + last_checked: SystemTime::now(), + }; + + self.hnsw_health.insert(index_name.to_string(), health.clone()); + health + } + + /// Check IVFFlat index health + pub fn check_ivfflat(&mut self, index_name: &str) -> IvfFlatHealth { + let health = IvfFlatHealth { + index_name: index_name.to_string(), + table_name: self.get_table_name(index_name), + cluster_balance: self.analyze_cluster_balance(index_name), + empty_clusters: self.count_empty_clusters(index_name), + centroid_quality: self.analyze_centroid_quality(index_name), + training_staleness_hours: self.get_training_staleness(index_name), + last_checked: SystemTime::now(), + }; + + self.ivfflat_health.insert(index_name.to_string(), health.clone()); + health + } + + /// Determine if index needs repair + pub fn needs_repair(&self, index_name: &str) -> Option { + if let Some(health) = self.hnsw_health.get(index_name) { + if health.fragmentation > 0.5 { + return Some(IndexIssue::Fragmentation { + index: index_name.to_string(), + level: health.fragmentation, + }); + } + if health.orphan_count > 100 { + return Some(IndexIssue::OrphanNodes { + index: index_name.to_string(), + count: health.orphan_count, + }); + } + if health.connectivity_score < 0.8 { + return Some(IndexIssue::PoorConnectivity { + index: index_name.to_string(), + score: health.connectivity_score, + }); + } + } + + if let Some(health) = self.ivfflat_health.get(index_name) { + if health.cluster_balance > 2.0 { // High std dev + return Some(IndexIssue::UnbalancedClusters { + index: index_name.to_string(), + balance: health.cluster_balance, + }); + } + if health.training_staleness_hours > 168.0 { // 1 week + return Some(IndexIssue::StaleTraining { + index: index_name.to_string(), + hours: health.training_staleness_hours, + }); + } + } + + None + } +} + +#[derive(Clone, Debug)] +pub enum IndexIssue { + Fragmentation { index: String, level: f32 }, + OrphanNodes { index: String, count: usize }, + PoorConnectivity { index: String, score: f32 }, + UnbalancedClusters { index: String, balance: f32 }, + StaleTraining { index: String, hours: f32 }, +} +``` + +### Learning Drift Detector + +```rust +pub struct LearningDriftDetector { + /// Pattern quality history + pattern_quality_history: RingBuffer, + + /// EWC task count + ewc_task_history: Vec, + + /// Drift threshold + drift_threshold: f32, + + /// Detected drifts + detected_drifts: Vec, +} + +impl LearningDriftDetector { + /// Detect learning drift + pub fn detect(&mut self, engine: &DagSonaEngine) -> Option { + let metrics = engine.metrics.to_json(); + + // Check pattern quality trend + let current_quality = self.compute_average_pattern_quality(engine); + self.pattern_quality_history.push(current_quality); + + if self.pattern_quality_history.len() >= 10 { + let trend = self.compute_trend(); + + if trend < -self.drift_threshold { + return Some(LearningDrift { + drift_type: DriftType::QualityDegradation, + severity: self.trend_to_severity(trend), + trend_value: trend, + recommendation: DriftRecommendation::ResetPatterns, + }); + } + } + + // Check EWC task explosion + let ewc_tasks = metrics["ewc_tasks"].as_u64().unwrap_or(0) as usize; + self.ewc_task_history.push(ewc_tasks); + + if ewc_tasks > 20 { + return Some(LearningDrift { + drift_type: DriftType::TaskExplosion, + severity: DriftSeverity::High, + trend_value: ewc_tasks as f32, + recommendation: DriftRecommendation::ConsolidateTasks, + }); + } + + // Check pattern staleness + let staleness = self.compute_pattern_staleness(engine); + if staleness > 0.8 { + return Some(LearningDrift { + drift_type: DriftType::PatternStaleness, + severity: DriftSeverity::Medium, + trend_value: staleness, + recommendation: DriftRecommendation::RefreshPatterns, + }); + } + + None + } + + fn compute_trend(&self) -> f32 { + let values: Vec = self.pattern_quality_history.iter().cloned().collect(); + if values.len() < 5 { + return 0.0; + } + + // Simple linear regression slope + let n = values.len() as f32; + let x_sum: f32 = (0..values.len()).map(|i| i as f32).sum(); + let y_sum: f32 = values.iter().sum(); + let xy_sum: f32 = values.iter().enumerate() + .map(|(i, &y)| i as f32 * y) + .sum(); + let x2_sum: f32 = (0..values.len()).map(|i| (i as f32).powi(2)).sum(); + + (n * xy_sum - x_sum * y_sum) / (n * x2_sum - x_sum.powi(2)) + } +} + +#[derive(Clone, Debug)] +pub struct LearningDrift { + pub drift_type: DriftType, + pub severity: DriftSeverity, + pub trend_value: f32, + pub recommendation: DriftRecommendation, +} + +#[derive(Clone, Debug)] +pub enum DriftType { + QualityDegradation, + TaskExplosion, + PatternStaleness, + DistributionShift, +} + +#[derive(Clone, Debug)] +pub enum DriftSeverity { + Low, + Medium, + High, +} + +#[derive(Clone, Debug)] +pub enum DriftRecommendation { + ResetPatterns, + ConsolidateTasks, + RefreshPatterns, + IncreaseLearningRate, + ReduceEwcLambda, +} +``` + +## Repair Strategies + +### Repair Strategy Trait + +```rust +pub trait RepairStrategy: Send + Sync { + /// Strategy identifier + fn name(&self) -> &str; + + /// Check if this strategy can handle the issue + fn can_repair(&self, issue: &Issue) -> bool; + + /// Execute repair + fn repair(&self, issue: &Issue, context: &RepairContext) -> Result; + + /// Validate repair success + fn validate(&self, issue: &Issue, result: &RepairResult) -> bool; + + /// Rollback repair + fn rollback(&self, result: &RepairResult) -> Result<(), RepairError>; + + /// Estimated repair time + fn estimated_duration(&self, issue: &Issue) -> Duration; +} +``` + +### Index Rebalance Strategy + +```rust +pub struct IndexRebalanceStrategy; + +impl RepairStrategy for IndexRebalanceStrategy { + fn name(&self) -> &str { + "index_rebalance" + } + + fn can_repair(&self, issue: &Issue) -> bool { + matches!(issue, + Issue::Index(IndexIssue::Fragmentation { .. }) | + Issue::Index(IndexIssue::OrphanNodes { .. }) | + Issue::Index(IndexIssue::UnbalancedClusters { .. }) + ) + } + + fn repair(&self, issue: &Issue, ctx: &RepairContext) -> Result { + match issue { + Issue::Index(IndexIssue::Fragmentation { index, level }) => { + if *level > 0.8 { + // Full rebuild required + self.rebuild_index(index, ctx) + } else { + // Partial rebalance + self.partial_rebalance(index, ctx) + } + } + + Issue::Index(IndexIssue::OrphanNodes { index, count }) => { + // Reconnect orphan nodes + self.reconnect_orphans(index, *count, ctx) + } + + Issue::Index(IndexIssue::UnbalancedClusters { index, .. }) => { + // Retrain clusters + self.retrain_clusters(index, ctx) + } + + _ => Err(RepairError::UnsupportedIssue), + } + } + + fn validate(&self, issue: &Issue, result: &RepairResult) -> bool { + // Re-check health after repair + match issue { + Issue::Index(idx_issue) => { + let health_checker = IndexHealthChecker::new(); + match idx_issue { + IndexIssue::Fragmentation { index, .. } => { + let health = health_checker.check_hnsw(index); + health.fragmentation < 0.3 + } + IndexIssue::OrphanNodes { index, .. } => { + let health = health_checker.check_hnsw(index); + health.orphan_count < 10 + } + _ => true, + } + } + _ => true, + } + } + + fn rollback(&self, result: &RepairResult) -> Result<(), RepairError> { + if let Some(backup) = &result.backup_data { + // Restore from backup + self.restore_backup(backup) + } else { + Ok(()) // No rollback needed + } + } + + fn estimated_duration(&self, issue: &Issue) -> Duration { + match issue { + Issue::Index(IndexIssue::Fragmentation { .. }) => Duration::from_secs(300), + Issue::Index(IndexIssue::OrphanNodes { count, .. }) => { + Duration::from_secs((*count / 100) as u64 + 10) + } + Issue::Index(IndexIssue::UnbalancedClusters { .. }) => Duration::from_secs(120), + _ => Duration::from_secs(60), + } + } +} + +impl IndexRebalanceStrategy { + fn rebuild_index(&self, index: &str, ctx: &RepairContext) -> Result { + // Create backup + let backup = self.backup_index(index)?; + + // Drop and recreate + ctx.execute_sql(&format!("REINDEX INDEX CONCURRENTLY {}", index))?; + + Ok(RepairResult { + success: true, + repair_type: "index_rebuild".to_string(), + duration: ctx.elapsed(), + backup_data: Some(backup), + details: json!({ + "index": index, + "method": "concurrent_reindex", + }), + }) + } + + fn partial_rebalance(&self, index: &str, ctx: &RepairContext) -> Result { + // Identify fragmented regions + let regions = self.identify_fragmented_regions(index)?; + + // Rebalance each region + for region in regions { + self.rebalance_region(index, ®ion, ctx)?; + } + + Ok(RepairResult { + success: true, + repair_type: "partial_rebalance".to_string(), + duration: ctx.elapsed(), + backup_data: None, + details: json!({ + "index": index, + "regions_rebalanced": regions.len(), + }), + }) + } + + fn reconnect_orphans(&self, index: &str, count: usize, ctx: &RepairContext) -> Result { + // Find orphan nodes + let orphans = self.find_orphan_nodes(index)?; + + // Reconnect each to nearest neighbors + let mut reconnected = 0; + for orphan in orphans { + if self.reconnect_node(index, orphan)? { + reconnected += 1; + } + } + + Ok(RepairResult { + success: reconnected > 0, + repair_type: "orphan_reconnection".to_string(), + duration: ctx.elapsed(), + backup_data: None, + details: json!({ + "index": index, + "total_orphans": count, + "reconnected": reconnected, + }), + }) + } +} +``` + +### Pattern Reset Strategy + +```rust +pub struct PatternResetStrategy; + +impl RepairStrategy for PatternResetStrategy { + fn name(&self) -> &str { + "pattern_reset" + } + + fn can_repair(&self, issue: &Issue) -> bool { + matches!(issue, + Issue::Learning(LearningDrift { drift_type: DriftType::QualityDegradation, .. }) | + Issue::Learning(LearningDrift { drift_type: DriftType::PatternStaleness, .. }) + ) + } + + fn repair(&self, issue: &Issue, ctx: &RepairContext) -> Result { + let engine = ctx.get_dag_engine()?; + + match issue { + Issue::Learning(drift) => { + match drift.recommendation { + DriftRecommendation::ResetPatterns => { + // Backup current patterns + let backup = self.backup_patterns(&engine)?; + + // Clear patterns but keep EWC state + { + let mut bank = engine.dag_reasoning_bank.write(); + bank.clear_patterns(); + } + + // Force immediate learning cycle + engine.run_background_cycle()?; + + Ok(RepairResult { + success: true, + repair_type: "pattern_reset".to_string(), + duration: ctx.elapsed(), + backup_data: Some(backup), + details: json!({ + "action": "reset_and_relearn", + }), + }) + } + + DriftRecommendation::RefreshPatterns => { + // Keep existing patterns, but force refresh + engine.run_background_cycle()?; + + // Consolidate similar patterns + { + let mut bank = engine.dag_reasoning_bank.write(); + bank.consolidate(0.9); + } + + Ok(RepairResult { + success: true, + repair_type: "pattern_refresh".to_string(), + duration: ctx.elapsed(), + backup_data: None, + details: json!({ + "action": "refresh_and_consolidate", + }), + }) + } + + _ => Err(RepairError::UnsupportedIssue), + } + } + _ => Err(RepairError::UnsupportedIssue), + } + } + + fn validate(&self, _issue: &Issue, _result: &RepairResult) -> bool { + // Validation will happen over time as new patterns are learned + true + } + + fn rollback(&self, result: &RepairResult) -> Result<(), RepairError> { + if let Some(backup) = &result.backup_data { + self.restore_patterns(backup) + } else { + Ok(()) + } + } + + fn estimated_duration(&self, _issue: &Issue) -> Duration { + Duration::from_secs(10) + } +} +``` + +## Healing Orchestration + +### Main Healing Loop + +```rust +impl SelfHealingEngine { + /// Run healing check cycle + pub fn run_check_cycle(&mut self) -> HealingCycleResult { + let start = Instant::now(); + let mut detected_issues = Vec::new(); + let mut repairs_initiated = Vec::new(); + + // 1. Anomaly detection + if let Some(obs) = self.collect_observation() { + let anomalies = self.anomaly_detector.detect(&obs); + for anomaly in anomalies { + detected_issues.push(Issue::Anomaly(anomaly)); + } + } + + // 2. Index health check + for index in self.get_monitored_indexes() { + if let Some(issue) = self.index_health_checker.needs_repair(&index) { + detected_issues.push(Issue::Index(issue)); + } + } + + // 3. Learning drift detection + for engine in self.get_dag_engines() { + if let Some(drift) = self.learning_drift_detector.detect(&engine) { + detected_issues.push(Issue::Learning(drift)); + } + } + + // 4. MinCut bottleneck check + for engine in self.get_dag_engines() { + if let Some(mincut) = &engine.mincut_engine { + let health = mincut.run_health_check(&engine.current_plan); + for alert in health.alerts { + if alert.severity >= AlertSeverity::Warning { + detected_issues.push(Issue::Bottleneck(alert)); + } + } + } + } + + // 5. Prioritize and diagnose + let prioritized = self.prioritize_issues(&detected_issues); + + // 6. Initiate repairs (if auto-heal enabled) + if self.config.auto_heal_enabled { + for issue in &prioritized { + if self.active_repairs.len() < self.config.max_concurrent_repairs { + if let Some(repair) = self.initiate_repair(issue) { + repairs_initiated.push(repair); + } + } + } + } + + // 7. Check active repairs + let completed_repairs = self.check_active_repairs(); + + HealingCycleResult { + detected_issues: detected_issues.len(), + repairs_initiated: repairs_initiated.len(), + repairs_completed: completed_repairs.len(), + active_repairs: self.active_repairs.len(), + duration: start.elapsed(), + } + } + + fn prioritize_issues(&self, issues: &[Issue]) -> Vec { + let mut prioritized = issues.to_vec(); + + prioritized.sort_by(|a, b| { + let a_priority = self.compute_priority(a); + let b_priority = self.compute_priority(b); + b_priority.cmp(&a_priority) + }); + + prioritized + } + + fn compute_priority(&self, issue: &Issue) -> u32 { + match issue { + Issue::Anomaly(a) => match a.severity { + AnomalySeverity::Critical => 100, + AnomalySeverity::High => 80, + AnomalySeverity::Medium => 50, + AnomalySeverity::Low => 20, + }, + Issue::Index(i) => match i { + IndexIssue::Fragmentation { level, .. } => (level * 100.0) as u32, + IndexIssue::OrphanNodes { count, .. } => (*count as u32).min(90), + _ => 50, + }, + Issue::Learning(d) => match d.severity { + DriftSeverity::High => 70, + DriftSeverity::Medium => 40, + DriftSeverity::Low => 20, + }, + Issue::Bottleneck(a) => match a.severity { + AlertSeverity::Critical => 90, + AlertSeverity::Warning => 60, + AlertSeverity::Info => 30, + }, + } + } + + fn initiate_repair(&mut self, issue: &Issue) -> Option { + // Find suitable strategy + let strategy = self.repair_strategies.iter() + .find(|s| s.can_repair(issue))?; + + let repair_id = self.generate_repair_id(); + + // Create repair context + let context = RepairContext::new(); + + // Start repair in background + let active_repair = ActiveRepair { + id: repair_id, + issue: issue.clone(), + strategy_name: strategy.name().to_string(), + started_at: Instant::now(), + status: RepairStatus::InProgress, + }; + + self.active_repairs.insert(repair_id, active_repair); + + // Execute repair + let result = strategy.repair(issue, &context); + + // Update status + if let Some(mut repair) = self.active_repairs.get_mut(&repair_id) { + repair.status = match result { + Ok(r) if r.success => RepairStatus::Completed(r), + Ok(r) => RepairStatus::Failed(RepairError::ValidationFailed), + Err(e) => RepairStatus::Failed(e), + }; + } + + Some(repair_id) + } +} +``` + +## SQL Interface + +```sql +-- Get healing status +SELECT ruvector_dag_healing_status(); + +-- Force health check +SELECT ruvector_dag_health_check('documents'); + +-- Get detected issues +SELECT * FROM ruvector_dag_detected_issues('documents'); + +-- Trigger manual repair +SELECT ruvector_dag_repair('documents', 'issue_id'); + +-- Get repair history +SELECT * FROM ruvector_dag_repair_history('documents', 7); -- Last 7 days + +-- Configure healing +SET ruvector.dag_healing_enabled = true; +SET ruvector.dag_healing_interval_ms = 300000; +SET ruvector.dag_healing_threshold = 0.3; +``` + +## Metrics and Monitoring + +```rust +#[derive(Clone, Debug, Default)] +pub struct HealingMetrics { + pub checks_performed: AtomicU64, + pub issues_detected: AtomicU64, + pub repairs_initiated: AtomicU64, + pub repairs_successful: AtomicU64, + pub repairs_failed: AtomicU64, + pub repairs_rolled_back: AtomicU64, + pub total_repair_time_ms: AtomicU64, + pub last_check_time: AtomicU64, +} + +impl HealingMetrics { + pub fn to_json(&self) -> serde_json::Value { + json!({ + "checks_performed": self.checks_performed.load(Ordering::Relaxed), + "issues_detected": self.issues_detected.load(Ordering::Relaxed), + "repairs_initiated": self.repairs_initiated.load(Ordering::Relaxed), + "repairs_successful": self.repairs_successful.load(Ordering::Relaxed), + "repairs_failed": self.repairs_failed.load(Ordering::Relaxed), + "repairs_rolled_back": self.repairs_rolled_back.load(Ordering::Relaxed), + "success_rate": self.success_rate(), + "avg_repair_time_ms": self.avg_repair_time(), + }) + } + + fn success_rate(&self) -> f64 { + let initiated = self.repairs_initiated.load(Ordering::Relaxed); + let successful = self.repairs_successful.load(Ordering::Relaxed); + if initiated > 0 { + successful as f64 / initiated as f64 + } else { + 1.0 + } + } +} +``` diff --git a/docs/dag/08-QUDAG-INTEGRATION.md b/docs/dag/08-QUDAG-INTEGRATION.md new file mode 100644 index 000000000..575a86168 --- /dev/null +++ b/docs/dag/08-QUDAG-INTEGRATION.md @@ -0,0 +1,739 @@ +# QuDAG Integration Specification + +## Overview + +This document specifies the optional integration between RuVector-Postgres Neural DAG system and QuDAG (Quantum-resistant Distributed DAG) for federated learning and distributed consensus on learned patterns. + +## Integration Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ QUDAG INTEGRATION LAYER │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌──────────────────────────────────────────────────────────────────────┐ │ +│ │ FEDERATED LEARNING │ │ +│ │ │ │ +│ │ Node A (US) Node B (EU) Node C (Asia) │ │ +│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ RuVector-PG │ │ RuVector-PG │ │ RuVector-PG │ │ │ +│ │ │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │ │ │ +│ │ │ │ Patterns │ │ │ │ Patterns │ │ │ │ Patterns │ │ │ │ +│ │ │ └────┬─────┘ │ │ └────┬─────┘ │ │ └────┬─────┘ │ │ │ +│ │ └──────┼───────┘ └──────┼───────┘ └──────┼───────┘ │ │ +│ │ │ │ │ │ │ +│ │ └────────────────────┼────────────────────┘ │ │ +│ │ ▼ │ │ +│ │ ┌─────────────────┐ │ │ +│ │ │ QuDAG Network │ │ │ +│ │ │ (QR-Avalanche) │ │ │ +│ │ └────────┬────────┘ │ │ +│ │ │ │ │ +│ │ ┌────────────────────┼────────────────────┐ │ │ +│ │ ▼ ▼ ▼ │ │ +│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │ +│ │ │ Consensus │ │ Consensus │ │ Consensus │ │ │ +│ │ │ Patterns │ │ Patterns │ │ Patterns │ │ │ +│ │ └────────────┘ └────────────┘ └────────────┘ │ │ +│ │ │ │ +│ └──────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌──────────────────────────────────────────────────────────────────────┐ │ +│ │ SECURITY LAYER │ │ +│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │ +│ │ │ ML-KEM │ │ ML-DSA │ │ Differential│ │ rUv │ │ │ +│ │ │ Encryption │ │ Signatures │ │ Privacy │ │ Tokens │ │ │ +│ │ └────────────┘ └────────────┘ └────────────┘ └────────────┘ │ │ +│ └──────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## QuDAG Client + +### Core Structure + +```rust +pub struct QuDagClient { + /// QuDAG node connection + node_url: String, + + /// Node identity (ML-DSA keypair) + identity: QuDagIdentity, + + /// Local pattern cache + pattern_cache: DashMap, + + /// Pending proposals + pending_proposals: DashMap, + + /// Configuration + config: QuDagConfig, + + /// Metrics + metrics: QuDagMetrics, +} + +#[derive(Clone)] +pub struct QuDagIdentity { + /// ML-DSA-65 public key + pub public_key: MlDsaPublicKey, + + /// ML-DSA-65 private key (encrypted at rest) + private_key: MlDsaPrivateKey, + + /// Node identifier + pub node_id: NodeId, + + /// Dark address (for anonymous communication) + pub dark_address: Option, +} + +#[derive(Clone, Debug)] +pub struct QuDagConfig { + /// Enable QuDAG integration + pub enabled: bool, + + /// QuDAG node URL + pub node_url: String, + + /// Differential privacy epsilon + pub dp_epsilon: f64, + + /// Minimum validators for consensus + pub min_validators: usize, + + /// Consensus timeout (seconds) + pub consensus_timeout_secs: u64, + + /// Sync interval (seconds) + pub sync_interval_secs: u64, + + /// Maximum patterns per proposal + pub max_patterns_per_proposal: usize, + + /// rUv staking requirement + pub min_stake_ruv: u64, +} + +impl Default for QuDagConfig { + fn default() -> Self { + Self { + enabled: false, + node_url: "https://yyz.qudag.darknet/mcp".to_string(), + dp_epsilon: 1.0, + min_validators: 5, + consensus_timeout_secs: 30, + sync_interval_secs: 3600, + max_patterns_per_proposal: 100, + min_stake_ruv: 10, + } + } +} +``` + +### Pattern Proposal + +```rust +impl QuDagClient { + /// Propose local patterns for consensus + pub async fn propose_patterns( + &self, + patterns: &[LearnedDagPattern], + ) -> Result { + // 1. Add differential privacy noise + let noisy_patterns = self.add_dp_noise(patterns)?; + + // 2. Create proposal + let proposal = PatternProposal { + id: self.generate_proposal_id(), + proposer: self.identity.node_id.clone(), + patterns: noisy_patterns, + stake: self.config.min_stake_ruv, + timestamp: SystemTime::now(), + signature: None, + }; + + // 3. Sign with ML-DSA + let signed_proposal = self.sign_proposal(proposal)?; + + // 4. Submit to QuDAG network + self.submit_proposal(&signed_proposal).await?; + + // 5. Track pending + self.pending_proposals.insert(signed_proposal.id, signed_proposal.clone()); + + Ok(signed_proposal.id) + } + + /// Add differential privacy noise to patterns + fn add_dp_noise(&self, patterns: &[LearnedDagPattern]) -> Result, QuDagError> { + let epsilon = self.config.dp_epsilon; + + patterns.iter() + .map(|p| { + // Add Laplace noise to centroid + let noisy_centroid: Vec = p.centroid.iter() + .map(|&v| { + let noise = laplace_sample(0.0, 1.0 / epsilon); + v + noise as f32 + }) + .collect(); + + // Quantize quality scores + let quantized_quality = (p.avg_metrics.quality * 10.0).round() / 10.0; + + Ok(NoisyPattern { + centroid: noisy_centroid, + attention_type: p.optimal_attention.clone(), + quality: quantized_quality, + sample_count_bucket: bucket_sample_count(p.sample_count), + }) + }) + .collect() + } + + /// Sign proposal with ML-DSA-65 + fn sign_proposal(&self, mut proposal: PatternProposal) -> Result { + let message = proposal.to_signing_bytes(); + let signature = self.identity.private_key.sign(&message)?; + proposal.signature = Some(signature); + Ok(proposal) + } + + /// Submit proposal to QuDAG network + async fn submit_proposal(&self, proposal: &PatternProposal) -> Result<(), QuDagError> { + // Connect to QuDAG MCP server + let client = McpClient::connect(&self.config.node_url).await?; + + // Call dag_submit tool + let response = client.call_tool("dag_submit", json!({ + "vertex_type": "pattern_proposal", + "payload": proposal.to_encrypted_bytes(&self.get_network_key())?, + "parents": self.get_recent_vertices().await?, + })).await?; + + if response["success"].as_bool().unwrap_or(false) { + Ok(()) + } else { + Err(QuDagError::SubmissionFailed( + response["error"].as_str().unwrap_or("Unknown error").to_string() + )) + } + } +} + +#[derive(Clone, Debug)] +pub struct PatternProposal { + pub id: ProposalId, + pub proposer: NodeId, + pub patterns: Vec, + pub stake: u64, + pub timestamp: SystemTime, + pub signature: Option, +} + +#[derive(Clone, Debug)] +pub struct NoisyPattern { + /// Centroid with DP noise + pub centroid: Vec, + + /// Attention type (no noise needed) + pub attention_type: DagAttentionType, + + /// Quantized quality + pub quality: f64, + + /// Bucketed sample count (privacy) + pub sample_count_bucket: SampleCountBucket, +} + +#[derive(Clone, Debug)] +pub enum SampleCountBucket { + Few, // < 10 + Some, // 10-50 + Many, // 50-200 + Lots, // > 200 +} +``` + +### Consensus Validation + +```rust +impl QuDagClient { + /// Validate incoming pattern proposals + pub async fn validate_proposal( + &self, + proposal: &PatternProposal, + ) -> Result { + // 1. Verify signature + if !self.verify_signature(proposal)? { + return Ok(ValidationResult::Rejected { + reason: "Invalid signature".to_string(), + }); + } + + // 2. Check stake + let balance = self.get_ruv_balance(&proposal.proposer).await?; + if balance < proposal.stake { + return Ok(ValidationResult::Rejected { + reason: "Insufficient stake".to_string(), + }); + } + + // 3. Validate pattern quality + let quality_scores: Vec = proposal.patterns.iter() + .map(|p| p.quality) + .collect(); + + let avg_quality = quality_scores.iter().sum::() / quality_scores.len() as f64; + if avg_quality < 0.3 { + return Ok(ValidationResult::Rejected { + reason: "Low quality patterns".to_string(), + }); + } + + // 4. Check for duplicate patterns + let duplicates = self.check_duplicates(&proposal.patterns).await?; + if duplicates > proposal.patterns.len() / 2 { + return Ok(ValidationResult::Rejected { + reason: "Too many duplicate patterns".to_string(), + }); + } + + // 5. Compute accuracy improvement (sample-based) + let improvement = self.estimate_improvement(&proposal.patterns).await?; + + Ok(ValidationResult::Accepted { + quality_score: avg_quality, + improvement_estimate: improvement, + validator: self.identity.node_id.clone(), + }) + } + + /// Submit validation to QuDAG + pub async fn submit_validation( + &self, + proposal_id: ProposalId, + result: &ValidationResult, + ) -> Result<(), QuDagError> { + let validation = Validation { + proposal_id, + result: result.clone(), + validator: self.identity.node_id.clone(), + timestamp: SystemTime::now(), + signature: None, + }; + + let signed = self.sign_validation(validation)?; + + let client = McpClient::connect(&self.config.node_url).await?; + client.call_tool("dag_submit", json!({ + "vertex_type": "pattern_validation", + "payload": signed.to_encrypted_bytes(&self.get_network_key())?, + "parents": [proposal_id.to_string()], + })).await?; + + Ok(()) + } +} + +#[derive(Clone, Debug)] +pub enum ValidationResult { + Accepted { + quality_score: f64, + improvement_estimate: f32, + validator: NodeId, + }, + Rejected { + reason: String, + }, +} +``` + +### Pattern Synchronization + +```rust +impl QuDagClient { + /// Sync consensus patterns from QuDAG + pub async fn sync_patterns(&self) -> Result { + let start = Instant::now(); + + // 1. Get latest consensus patterns + let client = McpClient::connect(&self.config.node_url).await?; + + let response = client.call_tool("dag_query", json!({ + "query_type": "consensus_patterns", + "since": self.last_sync_timestamp(), + "limit": 1000, + })).await?; + + let consensus_patterns: Vec = serde_json::from_value( + response["patterns"].clone() + )?; + + // 2. Verify signatures + let verified: Vec<_> = consensus_patterns.into_iter() + .filter(|p| self.verify_consensus_signature(p).unwrap_or(false)) + .collect(); + + // 3. Update local cache + let mut new_count = 0; + for pattern in &verified { + if !self.pattern_cache.contains_key(&pattern.id) { + self.pattern_cache.insert(pattern.id, pattern.clone()); + new_count += 1; + } + } + + // 4. Update local ReasoningBank + let imported = self.import_to_reasoning_bank(&verified)?; + + Ok(SyncResult { + patterns_received: verified.len(), + new_patterns: new_count, + patterns_imported: imported, + duration: start.elapsed(), + }) + } + + /// Import consensus patterns to local ReasoningBank + fn import_to_reasoning_bank(&self, patterns: &[ConsensusPattern]) -> Result { + let engines = get_all_dag_engines(); + let mut imported = 0; + + for pattern in patterns { + // Find matching local engine by pattern type + for engine in &engines { + let local_pattern = LearnedDagPattern { + id: self.generate_local_pattern_id(), + centroid: pattern.centroid.clone(), + optimal_params: ExecutionParams::default(), + optimal_attention: pattern.attention_type.clone(), + confidence: pattern.consensus_confidence, + sample_count: pattern.total_samples, + avg_metrics: AverageMetrics { + latency_us: 0.0, // Unknown from consensus + memory_bytes: 0.0, + quality: pattern.avg_quality, + }, + updated_at: SystemTime::now(), + }; + + let mut bank = engine.dag_reasoning_bank.write(); + bank.store(local_pattern); + imported += 1; + } + } + + Ok(imported) + } +} + +#[derive(Clone, Debug)] +pub struct ConsensusPattern { + pub id: PatternId, + pub centroid: Vec, + pub attention_type: DagAttentionType, + pub avg_quality: f64, + pub total_samples: usize, + pub consensus_confidence: f32, + pub validators: Vec, + pub signatures: Vec, + pub finalized_at: SystemTime, +} + +#[derive(Clone, Debug)] +pub struct SyncResult { + pub patterns_received: usize, + pub new_patterns: usize, + pub patterns_imported: usize, + pub duration: Duration, +} +``` + +## rUv Token Integration + +### Token Economy + +```rust +pub struct RuvTokenClient { + /// QuDAG client reference + qudag: Arc, + + /// Local balance cache + balance_cache: AtomicU64, + + /// Pending rewards + pending_rewards: DashMap, +} + +impl RuvTokenClient { + /// Check rUv balance + pub async fn get_balance(&self) -> Result { + let client = McpClient::connect(&self.qudag.config.node_url).await?; + + let response = client.call_tool("ruv_balance", json!({ + "address": self.qudag.identity.node_id.to_string(), + })).await?; + + let balance = response["balance"].as_u64().unwrap_or(0); + self.balance_cache.store(balance, Ordering::Relaxed); + + Ok(balance) + } + + /// Stake rUv for pattern proposal + pub async fn stake(&self, amount: u64) -> Result { + let client = McpClient::connect(&self.qudag.config.node_url).await?; + + let response = client.call_tool("ruv_stake", json!({ + "amount": amount, + "purpose": "pattern_proposal", + "signature": self.sign_stake_request(amount)?, + })).await?; + + Ok(TransactionId::from_str(response["tx_id"].as_str().unwrap())?) + } + + /// Claim rewards for accepted patterns + pub async fn claim_rewards(&self) -> Result { + let client = McpClient::connect(&self.qudag.config.node_url).await?; + + let response = client.call_tool("ruv_claim_rewards", json!({ + "address": self.qudag.identity.node_id.to_string(), + "signature": self.sign_claim_request()?, + })).await?; + + let claimed = response["claimed"].as_u64().unwrap_or(0); + let new_balance = response["new_balance"].as_u64().unwrap_or(0); + + self.balance_cache.store(new_balance, Ordering::Relaxed); + + Ok(ClaimResult { + amount_claimed: claimed, + new_balance, + }) + } +} + +/// Reward structure +#[derive(Clone, Debug)] +pub struct RewardStructure { + /// Base reward for accepted pattern + pub pattern_accepted: u64, // 10 rUv + + /// Bonus for accuracy improvement + pub accuracy_bonus_per_percent: u64, // 10 rUv per 1% + + /// Validation reward + pub validation_reward: u64, // 2 rUv + + /// Penalty for rejected pattern + pub rejection_penalty: u64, // 5 rUv + + /// Byzantine behavior penalty + pub byzantine_penalty: u64, // 1000 rUv +} + +impl Default for RewardStructure { + fn default() -> Self { + Self { + pattern_accepted: 10, + accuracy_bonus_per_percent: 10, + validation_reward: 2, + rejection_penalty: 5, + byzantine_penalty: 1000, + } + } +} +``` + +## Security Layer + +### ML-KEM Encryption + +```rust +pub struct PatternEncryption { + /// Network public key (for encryption) + network_key: MlKemPublicKey, + + /// Local private key (for decryption) + local_key: MlKemPrivateKey, +} + +impl PatternEncryption { + /// Encrypt pattern for network transmission + pub fn encrypt(&self, pattern: &NoisyPattern) -> Result { + let plaintext = pattern.to_bytes(); + + // Encapsulate shared secret + let (ciphertext, shared_secret) = self.network_key.encapsulate()?; + + // Derive key from shared secret + let key = blake3::derive_key("QuDAG Pattern Encryption", &shared_secret); + + // Encrypt with ChaCha20-Poly1305 + let nonce = generate_nonce(); + let encrypted = chacha20_poly1305_encrypt(&key, &nonce, &plaintext)?; + + Ok(EncryptedPattern { + ciphertext, + encrypted_data: encrypted, + nonce, + }) + } + + /// Decrypt pattern from network + pub fn decrypt(&self, encrypted: &EncryptedPattern) -> Result { + // Decapsulate shared secret + let shared_secret = self.local_key.decapsulate(&encrypted.ciphertext)?; + + // Derive key + let key = blake3::derive_key("QuDAG Pattern Encryption", &shared_secret); + + // Decrypt + let plaintext = chacha20_poly1305_decrypt( + &key, + &encrypted.nonce, + &encrypted.encrypted_data, + )?; + + NoisyPattern::from_bytes(&plaintext) + } +} +``` + +### ML-DSA Signatures + +```rust +pub struct PatternSigning { + /// Signing key + private_key: MlDsaPrivateKey, + + /// Verification key + public_key: MlDsaPublicKey, +} + +impl PatternSigning { + /// Sign pattern proposal + pub fn sign_proposal(&self, proposal: &PatternProposal) -> Result { + let message = proposal.to_signing_bytes(); + self.private_key.sign(&message) + } + + /// Verify proposal signature + pub fn verify_proposal( + &self, + proposal: &PatternProposal, + public_key: &MlDsaPublicKey, + ) -> Result { + let message = proposal.to_signing_bytes(); + let signature = proposal.signature.as_ref() + .ok_or(CryptoError::MissingSignature)?; + + public_key.verify(&message, signature) + } + + /// Sign validation + pub fn sign_validation(&self, validation: &Validation) -> Result { + let message = validation.to_signing_bytes(); + self.private_key.sign(&message) + } +} +``` + +## SQL Interface + +```sql +-- Enable QuDAG integration +SELECT ruvector_dag_qudag_enable('{ + "node_url": "https://yyz.qudag.darknet/mcp", + "dp_epsilon": 1.0, + "min_stake_ruv": 10 +}'::jsonb); + +-- Register identity +SELECT ruvector_dag_qudag_register(); + +-- Propose patterns for consensus +SELECT ruvector_dag_qudag_propose('documents'); + +-- Sync consensus patterns +SELECT ruvector_dag_qudag_sync(); + +-- Get rUv balance +SELECT ruvector_dag_ruv_balance(); + +-- Claim rewards +SELECT ruvector_dag_ruv_claim(); + +-- Get QuDAG status +SELECT ruvector_dag_qudag_status(); +``` + +## Configuration + +### PostgreSQL GUC Variables + +```sql +-- Enable/disable QuDAG +SET ruvector.dag_qudag_enabled = true; + +-- QuDAG node URL +SET ruvector.dag_qudag_node_url = 'https://yyz.qudag.darknet/mcp'; + +-- Differential privacy epsilon +SET ruvector.dag_qudag_dp_epsilon = 1.0; + +-- Sync interval (seconds) +SET ruvector.dag_qudag_sync_interval = 3600; + +-- Minimum stake for proposals +SET ruvector.dag_qudag_min_stake = 10; +``` + +## Metrics + +```rust +#[derive(Clone, Debug, Default)] +pub struct QuDagMetrics { + pub proposals_submitted: AtomicU64, + pub proposals_accepted: AtomicU64, + pub proposals_rejected: AtomicU64, + pub validations_performed: AtomicU64, + pub patterns_synced: AtomicU64, + pub ruv_earned: AtomicU64, + pub ruv_spent: AtomicU64, + pub last_sync_time: AtomicU64, +} + +impl QuDagMetrics { + pub fn to_json(&self) -> serde_json::Value { + json!({ + "proposals_submitted": self.proposals_submitted.load(Ordering::Relaxed), + "proposals_accepted": self.proposals_accepted.load(Ordering::Relaxed), + "proposals_rejected": self.proposals_rejected.load(Ordering::Relaxed), + "acceptance_rate": self.acceptance_rate(), + "validations_performed": self.validations_performed.load(Ordering::Relaxed), + "patterns_synced": self.patterns_synced.load(Ordering::Relaxed), + "ruv_net": self.ruv_net(), + }) + } + + fn acceptance_rate(&self) -> f64 { + let submitted = self.proposals_submitted.load(Ordering::Relaxed); + let accepted = self.proposals_accepted.load(Ordering::Relaxed); + if submitted > 0 { + accepted as f64 / submitted as f64 + } else { + 0.0 + } + } + + fn ruv_net(&self) -> i64 { + self.ruv_earned.load(Ordering::Relaxed) as i64 + - self.ruv_spent.load(Ordering::Relaxed) as i64 + } +} +``` diff --git a/docs/dag/09-SQL-API.md b/docs/dag/09-SQL-API.md new file mode 100644 index 000000000..0841f6b31 --- /dev/null +++ b/docs/dag/09-SQL-API.md @@ -0,0 +1,790 @@ +# SQL API Reference + +## Overview + +Complete SQL API for the Neural DAG Learning system integrated with RuVector-Postgres. + +## Configuration Functions + +### System Configuration + +```sql +-- Enable/disable neural DAG learning +SELECT ruvector.dag_set_enabled(enabled BOOLEAN) RETURNS VOID; + +-- Configure learning rate +SELECT ruvector.dag_set_learning_rate(rate FLOAT8) RETURNS VOID; + +-- Set attention mechanism +SELECT ruvector.dag_set_attention( + mechanism TEXT -- 'topological', 'causal_cone', 'critical_path', + -- 'mincut_gated', 'hierarchical_lorentz', + -- 'parallel_branch', 'temporal_btsp', 'auto' +) RETURNS VOID; + +-- Configure SONA parameters +SELECT ruvector.dag_configure_sona( + micro_lora_rank INT DEFAULT 2, + base_lora_rank INT DEFAULT 8, + ewc_lambda FLOAT8 DEFAULT 5000.0, + pattern_clusters INT DEFAULT 100 +) RETURNS VOID; + +-- Set QuDAG network endpoint +SELECT ruvector.dag_set_qudag_endpoint( + endpoint TEXT, + stake_amount FLOAT8 DEFAULT 0.0 +) RETURNS VOID; +``` + +### Runtime Status + +```sql +-- Get current configuration +SELECT * FROM ruvector.dag_config(); +-- Returns: (enabled, learning_rate, attention_mechanism, +-- micro_lora_rank, base_lora_rank, ewc_lambda, qudag_endpoint) + +-- Get system status +SELECT * FROM ruvector.dag_status(); +-- Returns: (active_patterns, total_trajectories, avg_improvement, +-- attention_hits, learning_rate_effective, qudag_connected) + +-- Check health +SELECT * FROM ruvector.dag_health_check(); +-- Returns: (component, status, last_check, message) +``` + +## Query Analysis Functions + +### Plan Analysis + +```sql +-- Analyze query plan and return neural DAG insights +SELECT * FROM ruvector.dag_analyze_plan( + query_text TEXT +) RETURNS TABLE ( + node_id INT, + operator_type TEXT, + criticality FLOAT8, + bottleneck_score FLOAT8, + embedding VECTOR(256), + parent_ids INT[], + child_ids INT[], + estimated_cost FLOAT8, + recommendations TEXT[] +); + +-- Get critical path for query +SELECT * FROM ruvector.dag_critical_path( + query_text TEXT +) RETURNS TABLE ( + path_position INT, + node_id INT, + operator_type TEXT, + accumulated_cost FLOAT8, + attention_weight FLOAT8 +); + +-- Identify bottlenecks +SELECT * FROM ruvector.dag_bottlenecks( + query_text TEXT, + threshold FLOAT8 DEFAULT 0.7 +) RETURNS TABLE ( + node_id INT, + operator_type TEXT, + bottleneck_score FLOAT8, + impact_estimate FLOAT8, + suggested_action TEXT +); + +-- Get min-cut analysis +SELECT * FROM ruvector.dag_mincut_analysis( + query_text TEXT +) RETURNS TABLE ( + cut_id INT, + source_nodes INT[], + sink_nodes INT[], + cut_capacity FLOAT8, + parallelization_opportunity BOOLEAN +); +``` + +### Query Optimization + +```sql +-- Get optimization suggestions +SELECT * FROM ruvector.dag_suggest_optimizations( + query_text TEXT +) RETURNS TABLE ( + suggestion_id INT, + category TEXT, -- 'index', 'join_order', 'parallelism', 'memory' + description TEXT, + expected_improvement FLOAT8, + implementation_sql TEXT, + confidence FLOAT8 +); + +-- Rewrite query using learned patterns +SELECT ruvector.dag_rewrite_query( + query_text TEXT +) RETURNS TEXT; + +-- Estimate query with neural predictions +SELECT * FROM ruvector.dag_estimate( + query_text TEXT +) RETURNS TABLE ( + metric TEXT, + postgres_estimate FLOAT8, + neural_estimate FLOAT8, + confidence FLOAT8 +); +``` + +## Attention Mechanism Functions + +### Attention Scores + +```sql +-- Compute attention for query DAG +SELECT * FROM ruvector.dag_attention_scores( + query_text TEXT, + mechanism TEXT DEFAULT 'auto' +) RETURNS TABLE ( + node_id INT, + attention_weight FLOAT8, + query_contribution FLOAT8[], + key_contribution FLOAT8[] +); + +-- Get attention matrix +SELECT ruvector.dag_attention_matrix( + query_text TEXT, + mechanism TEXT DEFAULT 'auto' +) RETURNS FLOAT8[][]; + +-- Visualize attention (returns DOT graph) +SELECT ruvector.dag_attention_visualize( + query_text TEXT, + mechanism TEXT DEFAULT 'auto', + format TEXT DEFAULT 'dot' -- 'dot', 'json', 'ascii' +) RETURNS TEXT; +``` + +### Attention Configuration + +```sql +-- Set attention hyperparameters +SELECT ruvector.dag_attention_configure( + mechanism TEXT, + params JSONB + -- Example params: + -- topological: {"max_depth": 5, "decay_factor": 0.9} + -- causal_cone: {"time_window": 1000, "future_discount": 0.5} + -- critical_path: {"path_weight": 2.0, "branch_penalty": 0.3} + -- mincut_gated: {"gate_threshold": 0.1, "flow_capacity": "cost"} + -- hierarchical_lorentz: {"curvature": -1.0, "time_scale": 0.1} + -- parallel_branch: {"max_branches": 8, "sync_penalty": 0.2} + -- temporal_btsp: {"plateau_duration": 100, "eligibility_decay": 0.95} +) RETURNS VOID; + +-- Get attention statistics +SELECT * FROM ruvector.dag_attention_stats() +RETURNS TABLE ( + mechanism TEXT, + invocations BIGINT, + avg_latency_us FLOAT8, + hit_rate FLOAT8, + improvement_ratio FLOAT8 +); +``` + +## SONA Learning Functions + +### Pattern Management + +```sql +-- Store a learned pattern +SELECT ruvector.dag_store_pattern( + pattern_vector VECTOR(256), + pattern_metadata JSONB, + quality_score FLOAT8 +) RETURNS BIGINT; -- pattern_id + +-- Query similar patterns +SELECT * FROM ruvector.dag_query_patterns( + query_vector VECTOR(256), + k INT DEFAULT 5, + similarity_threshold FLOAT8 DEFAULT 0.7 +) RETURNS TABLE ( + pattern_id BIGINT, + similarity FLOAT8, + quality_score FLOAT8, + metadata JSONB, + usage_count INT, + last_used TIMESTAMPTZ +); + +-- Get pattern clusters (ReasoningBank) +SELECT * FROM ruvector.dag_pattern_clusters() +RETURNS TABLE ( + cluster_id INT, + centroid VECTOR(256), + member_count INT, + avg_quality FLOAT8, + representative_query TEXT +); + +-- Force pattern consolidation +SELECT ruvector.dag_consolidate_patterns( + target_clusters INT DEFAULT 100 +) RETURNS TABLE ( + clusters_before INT, + clusters_after INT, + patterns_merged INT, + consolidation_time_ms FLOAT8 +); +``` + +### Trajectory Management + +```sql +-- Record a learning trajectory +SELECT ruvector.dag_record_trajectory( + query_hash BIGINT, + dag_structure JSONB, + execution_time_ms FLOAT8, + improvement_ratio FLOAT8, + attention_mechanism TEXT +) RETURNS BIGINT; -- trajectory_id + +-- Get trajectory history +SELECT * FROM ruvector.dag_trajectory_history( + time_range TSTZRANGE DEFAULT NULL, + min_improvement FLOAT8 DEFAULT 0.0, + limit_count INT DEFAULT 100 +) RETURNS TABLE ( + trajectory_id BIGINT, + query_hash BIGINT, + recorded_at TIMESTAMPTZ, + execution_time_ms FLOAT8, + improvement_ratio FLOAT8, + attention_mechanism TEXT +); + +-- Analyze trajectory trends +SELECT * FROM ruvector.dag_trajectory_trends( + window_size INTERVAL DEFAULT '1 hour' +) RETURNS TABLE ( + window_start TIMESTAMPTZ, + trajectory_count INT, + avg_improvement FLOAT8, + best_mechanism TEXT, + pattern_discoveries INT +); +``` + +### Learning Control + +```sql +-- Trigger immediate learning cycle +SELECT ruvector.dag_learn_now() RETURNS TABLE ( + patterns_updated INT, + new_clusters INT, + ewc_constraints_updated INT, + cycle_time_ms FLOAT8 +); + +-- Reset learning state (use with caution) +SELECT ruvector.dag_reset_learning( + preserve_patterns BOOLEAN DEFAULT TRUE, + preserve_trajectories BOOLEAN DEFAULT FALSE +) RETURNS VOID; + +-- Export learned state +SELECT ruvector.dag_export_state() RETURNS BYTEA; + +-- Import learned state +SELECT ruvector.dag_import_state(state_data BYTEA) RETURNS TABLE ( + patterns_imported INT, + trajectories_imported INT, + clusters_restored INT +); + +-- Get EWC constraint info +SELECT * FROM ruvector.dag_ewc_constraints() +RETURNS TABLE ( + parameter_name TEXT, + fisher_importance FLOAT8, + optimal_value FLOAT8, + last_updated TIMESTAMPTZ +); +``` + +## Self-Healing Functions + +### Health Monitoring + +```sql +-- Run comprehensive health check +SELECT * FROM ruvector.dag_health_report() +RETURNS TABLE ( + subsystem TEXT, + status TEXT, + score FLOAT8, + issues TEXT[], + recommendations TEXT[] +); + +-- Get anomaly detection results +SELECT * FROM ruvector.dag_anomalies( + time_range TSTZRANGE DEFAULT '[now - 1 hour, now]'::TSTZRANGE +) RETURNS TABLE ( + anomaly_id BIGINT, + detected_at TIMESTAMPTZ, + anomaly_type TEXT, + severity TEXT, + affected_component TEXT, + z_score FLOAT8, + resolved BOOLEAN +); + +-- Check index health +SELECT * FROM ruvector.dag_index_health() +RETURNS TABLE ( + index_name TEXT, + index_type TEXT, + fragmentation FLOAT8, + recall_estimate FLOAT8, + recommended_action TEXT +); + +-- Check learning drift +SELECT * FROM ruvector.dag_learning_drift() +RETURNS TABLE ( + metric TEXT, + current_value FLOAT8, + baseline_value FLOAT8, + drift_magnitude FLOAT8, + trend TEXT +); +``` + +### Repair Operations + +```sql +-- Trigger automatic repair +SELECT * FROM ruvector.dag_auto_repair() +RETURNS TABLE ( + repair_id BIGINT, + repair_type TEXT, + target TEXT, + status TEXT, + duration_ms FLOAT8 +); + +-- Rebalance specific index +SELECT ruvector.dag_rebalance_index( + index_name TEXT, + target_recall FLOAT8 DEFAULT 0.95 +) RETURNS TABLE ( + vectors_moved INT, + new_recall FLOAT8, + duration_ms FLOAT8 +); + +-- Reset pattern quality scores +SELECT ruvector.dag_reset_pattern_quality( + pattern_ids BIGINT[] DEFAULT NULL -- NULL = all patterns +) RETURNS INT; -- patterns reset + +-- Force cluster recomputation +SELECT ruvector.dag_recompute_clusters( + algorithm TEXT DEFAULT 'kmeans_pp' +) RETURNS TABLE ( + old_clusters INT, + new_clusters INT, + silhouette_score FLOAT8 +); +``` + +## QuDAG Integration Functions + +### Network Operations + +```sql +-- Connect to QuDAG network +SELECT ruvector.qudag_connect( + endpoint TEXT, + identity_key BYTEA DEFAULT NULL -- auto-generate if NULL +) RETURNS TABLE ( + connected BOOLEAN, + node_id TEXT, + network_version TEXT +); + +-- Get network status +SELECT * FROM ruvector.qudag_status() +RETURNS TABLE ( + connected BOOLEAN, + node_id TEXT, + peers INT, + latest_round BIGINT, + sync_status TEXT +); + +-- Propose pattern to network +SELECT ruvector.qudag_propose_pattern( + pattern_vector VECTOR(256), + metadata JSONB, + stake_amount FLOAT8 DEFAULT 0.0 +) RETURNS TABLE ( + proposal_id TEXT, + submitted_at TIMESTAMPTZ, + status TEXT +); + +-- Check proposal status +SELECT * FROM ruvector.qudag_proposal_status( + proposal_id TEXT +) RETURNS TABLE ( + status TEXT, + votes_for INT, + votes_against INT, + finalized BOOLEAN, + finalized_at TIMESTAMPTZ +); + +-- Sync patterns from network +SELECT * FROM ruvector.qudag_sync_patterns( + since_round BIGINT DEFAULT 0 +) RETURNS TABLE ( + patterns_received INT, + patterns_applied INT, + conflicts_resolved INT +); +``` + +### Token Operations + +```sql +-- Get rUv balance +SELECT ruvector.qudag_balance() RETURNS FLOAT8; + +-- Stake tokens +SELECT ruvector.qudag_stake( + amount FLOAT8 +) RETURNS TABLE ( + new_stake FLOAT8, + tx_hash TEXT +); + +-- Claim rewards +SELECT * FROM ruvector.qudag_claim_rewards() +RETURNS TABLE ( + amount FLOAT8, + tx_hash TEXT, + source TEXT +); + +-- Get staking info +SELECT * FROM ruvector.qudag_staking_info() +RETURNS TABLE ( + staked_amount FLOAT8, + pending_rewards FLOAT8, + lock_until TIMESTAMPTZ, + apr_estimate FLOAT8 +); +``` + +### Cryptographic Operations + +```sql +-- Generate ML-KEM keypair +SELECT ruvector.qudag_generate_kem_keypair() +RETURNS TABLE ( + public_key BYTEA, + secret_key_id TEXT -- stored securely +); + +-- Encrypt data for peer +SELECT ruvector.qudag_encrypt( + plaintext BYTEA, + recipient_pubkey BYTEA +) RETURNS TABLE ( + ciphertext BYTEA, + encapsulated_key BYTEA +); + +-- Decrypt received data +SELECT ruvector.qudag_decrypt( + ciphertext BYTEA, + encapsulated_key BYTEA, + secret_key_id TEXT +) RETURNS BYTEA; + +-- Sign data +SELECT ruvector.qudag_sign( + data BYTEA +) RETURNS BYTEA; -- ML-DSA signature + +-- Verify signature +SELECT ruvector.qudag_verify( + data BYTEA, + signature BYTEA, + pubkey BYTEA +) RETURNS BOOLEAN; +``` + +## Monitoring and Statistics + +### Performance Metrics + +```sql +-- Get overall statistics +SELECT * FROM ruvector.dag_statistics() +RETURNS TABLE ( + metric TEXT, + value FLOAT8, + unit TEXT, + updated_at TIMESTAMPTZ +); + +-- Get latency breakdown +SELECT * FROM ruvector.dag_latency_breakdown( + time_range TSTZRANGE DEFAULT '[now - 1 hour, now]'::TSTZRANGE +) RETURNS TABLE ( + component TEXT, + p50_us FLOAT8, + p95_us FLOAT8, + p99_us FLOAT8, + max_us FLOAT8 +); + +-- Get memory usage +SELECT * FROM ruvector.dag_memory_usage() +RETURNS TABLE ( + component TEXT, + allocated_bytes BIGINT, + used_bytes BIGINT, + peak_bytes BIGINT +); + +-- Get throughput metrics +SELECT * FROM ruvector.dag_throughput( + window INTERVAL DEFAULT '1 minute' +) RETURNS TABLE ( + metric TEXT, + count BIGINT, + per_second FLOAT8 +); +``` + +### Debugging + +```sql +-- Enable debug logging +SELECT ruvector.dag_set_debug( + enabled BOOLEAN, + components TEXT[] DEFAULT ARRAY['all'] +) RETURNS VOID; + +-- Get recent debug logs +SELECT * FROM ruvector.dag_debug_logs( + since TIMESTAMPTZ DEFAULT now() - interval '5 minutes', + component TEXT DEFAULT NULL, + severity TEXT DEFAULT NULL -- 'debug', 'info', 'warn', 'error' +) RETURNS TABLE ( + logged_at TIMESTAMPTZ, + component TEXT, + severity TEXT, + message TEXT, + context JSONB +); + +-- Trace single query +SELECT * FROM ruvector.dag_trace_query( + query_text TEXT +) RETURNS TABLE ( + step INT, + operation TEXT, + duration_us FLOAT8, + details JSONB +); + +-- Export diagnostics bundle +SELECT ruvector.dag_export_diagnostics() RETURNS BYTEA; +``` + +## Batch Operations + +### Bulk Processing + +```sql +-- Analyze multiple queries +SELECT * FROM ruvector.dag_bulk_analyze( + queries TEXT[] +) RETURNS TABLE ( + query_index INT, + bottleneck_count INT, + suggested_mechanism TEXT, + estimated_improvement FLOAT8 +); + +-- Pre-warm patterns for workload +SELECT ruvector.dag_prewarm_patterns( + representative_queries TEXT[] +) RETURNS TABLE ( + patterns_loaded INT, + cache_hit_rate FLOAT8 +); + +-- Batch record trajectories +SELECT ruvector.dag_bulk_record_trajectories( + trajectories JSONB[] +) RETURNS INT; -- trajectories recorded +``` + +## Views + +### System Views + +```sql +-- Active configuration +CREATE VIEW ruvector.dag_active_config AS +SELECT * FROM ruvector.dag_config(); + +-- Recent patterns +CREATE VIEW ruvector.dag_recent_patterns AS +SELECT pattern_id, created_at, quality_score, usage_count +FROM ruvector.dag_patterns +WHERE created_at > now() - interval '24 hours' +ORDER BY quality_score DESC; + +-- Attention effectiveness +CREATE VIEW ruvector.dag_attention_effectiveness AS +SELECT + mechanism, + count(*) as uses, + avg(improvement_ratio) as avg_improvement, + percentile_cont(0.95) WITHIN GROUP (ORDER BY improvement_ratio) as p95_improvement +FROM ruvector.dag_trajectories +WHERE recorded_at > now() - interval '7 days' +GROUP BY mechanism; + +-- Health summary +CREATE VIEW ruvector.dag_health_summary AS +SELECT + subsystem, + status, + score, + array_length(issues, 1) as issue_count +FROM ruvector.dag_health_report(); +``` + +## Installation SQL + +```sql +-- Create extension +CREATE EXTENSION IF NOT EXISTS ruvector_dag CASCADE; + +-- Required tables (auto-created by extension) +-- ruvector.dag_patterns - Learned patterns storage +-- ruvector.dag_trajectories - Learning trajectory history +-- ruvector.dag_clusters - Pattern clusters (ReasoningBank) +-- ruvector.dag_anomalies - Detected anomalies log +-- ruvector.dag_repairs - Repair history +-- ruvector.dag_qudag_proposals - QuDAG proposal tracking + +-- Recommended indexes +CREATE INDEX ON ruvector.dag_patterns USING hnsw (pattern_vector vector_cosine_ops); +CREATE INDEX ON ruvector.dag_trajectories (recorded_at DESC); +CREATE INDEX ON ruvector.dag_trajectories (query_hash); +CREATE INDEX ON ruvector.dag_anomalies (detected_at DESC) WHERE NOT resolved; + +-- Grant permissions +GRANT USAGE ON SCHEMA ruvector TO PUBLIC; +GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA ruvector TO PUBLIC; +GRANT SELECT ON ALL TABLES IN SCHEMA ruvector TO PUBLIC; +``` + +## Usage Examples + +### Basic Query Optimization + +```sql +-- Enable neural DAG learning +SELECT ruvector.dag_set_enabled(true); + +-- Analyze a query +SELECT * FROM ruvector.dag_analyze_plan($$ + SELECT v.*, m.category + FROM vectors v + JOIN metadata m ON v.id = m.vector_id + WHERE v.embedding <-> $1 < 0.5 + ORDER BY v.embedding <-> $1 + LIMIT 100 +$$); + +-- Get optimization suggestions +SELECT * FROM ruvector.dag_suggest_optimizations($$ + SELECT v.*, m.category + FROM vectors v + JOIN metadata m ON v.id = m.vector_id + WHERE v.embedding <-> $1 < 0.5 + ORDER BY v.embedding <-> $1 + LIMIT 100 +$$); +``` + +### Attention Mechanism Selection + +```sql +-- Let system choose best attention +SELECT ruvector.dag_set_attention('auto'); + +-- Or manually select based on workload +-- For deep query plans: +SELECT ruvector.dag_set_attention('topological'); + +-- For time-series workloads: +SELECT ruvector.dag_set_attention('causal_cone'); + +-- For CPU-bound queries: +SELECT ruvector.dag_set_attention('critical_path'); +``` + +### Distributed Learning with QuDAG + +```sql +-- Connect to QuDAG network +SELECT * FROM ruvector.qudag_connect( + 'https://qudag.example.com:8443' +); + +-- Stake tokens for participation +SELECT * FROM ruvector.qudag_stake(100.0); + +-- Patterns are now automatically shared and validated +-- Check sync status +SELECT * FROM ruvector.qudag_status(); +``` + +## Error Codes + +| Code | Name | Description | +|------|------|-------------| +| RV001 | DAG_DISABLED | Neural DAG learning is disabled | +| RV002 | INVALID_ATTENTION | Unknown attention mechanism | +| RV003 | PATTERN_NOT_FOUND | Referenced pattern does not exist | +| RV004 | LEARNING_FAILED | Learning cycle failed | +| RV005 | QUDAG_DISCONNECTED | Not connected to QuDAG network | +| RV006 | QUDAG_AUTH_FAILED | QuDAG authentication failed | +| RV007 | INSUFFICIENT_STAKE | Not enough staked tokens | +| RV008 | CRYPTO_ERROR | Cryptographic operation failed | +| RV009 | REPAIR_FAILED | Self-healing repair failed | +| RV010 | TRAJECTORY_OVERFLOW | Trajectory buffer full | + +--- + +*Document: 09-SQL-API.md | Version: 1.0 | Last Updated: 2025-01-XX* diff --git a/docs/dag/10-TESTING-STRATEGY.md b/docs/dag/10-TESTING-STRATEGY.md new file mode 100644 index 000000000..eab32dd70 --- /dev/null +++ b/docs/dag/10-TESTING-STRATEGY.md @@ -0,0 +1,1013 @@ +# Testing Strategy + +## Overview + +Comprehensive testing strategy for the Neural DAG Learning system, ensuring correctness, performance, and reliability across all components. + +## Testing Layers + +``` +┌─────────────────────────────────────────────────────────────┐ +│ End-to-End Tests │ +│ (Full system integration) │ +├─────────────────────────────────────────────────────────────┤ +│ Integration Tests │ +│ (Component interaction testing) │ +├─────────────────────────────────────────────────────────────┤ +│ Unit Tests │ +│ (Individual function/module) │ +├─────────────────────────────────────────────────────────────┤ +│ Property Tests │ +│ (Invariant verification) │ +├─────────────────────────────────────────────────────────────┤ +│ Benchmark Tests │ +│ (Performance validation) │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Test Categories + +### 1. Unit Tests + +#### DAG Attention Mechanisms + +```rust +// tests/unit/attention/mod.rs + +#[cfg(test)] +mod topological_attention_tests { + use super::*; + use ruvector_dag::attention::TopologicalAttention; + + #[test] + fn test_topological_sort_simple_dag() { + let mut dag = QueryDag::new(); + dag.add_node(0, OperatorNode::seq_scan("users")); + dag.add_node(1, OperatorNode::index_scan("idx_users_email")); + dag.add_node(2, OperatorNode::hash_join()); + dag.add_edge(0, 2); + dag.add_edge(1, 2); + + let attention = TopologicalAttention::new(TopologicalConfig::default()); + let scores = attention.forward(&dag).unwrap(); + + // Root nodes should have highest attention + assert!(scores[&2] > scores[&0]); + assert!(scores[&2] > scores[&1]); + } + + #[test] + fn test_topological_attention_decay() { + let config = TopologicalConfig { + decay_factor: 0.5, + max_depth: 10, + }; + let attention = TopologicalAttention::new(config); + + // Deep DAG test + let dag = create_linear_dag(10); + let scores = attention.forward(&dag).unwrap(); + + // Verify decay: score[depth] ≈ base * decay^depth + for depth in 1..10 { + let ratio = scores[&depth] / scores[&(depth - 1)]; + assert!((ratio - 0.5).abs() < 0.01); + } + } + + #[test] + fn test_topological_handles_cycles_gracefully() { + // This should not happen in query DAGs, but test robustness + let dag = create_dag_with_back_edge(); + let attention = TopologicalAttention::new(TopologicalConfig::default()); + + let result = attention.forward(&dag); + assert!(result.is_err()); + assert!(matches!(result.unwrap_err(), AttentionError::CycleDetected)); + } +} + +#[cfg(test)] +mod causal_cone_attention_tests { + #[test] + fn test_causal_cone_future_discount() { + let config = CausalConeConfig { + future_discount: 0.5, + time_window_ms: 1000, + }; + let attention = CausalConeAttention::new(config); + + let dag = create_temporal_dag(); + let scores = attention.forward(&dag).unwrap(); + + // Future operators should be discounted + assert!(scores[&"past_op"] > scores[&"future_op"]); + } + + #[test] + fn test_causal_cone_respects_time_window() { + let config = CausalConeConfig { + future_discount: 0.5, + time_window_ms: 100, + }; + let attention = CausalConeAttention::new(config); + + let dag = create_wide_time_range_dag(); + let scores = attention.forward(&dag).unwrap(); + + // Operators outside time window should have near-zero attention + assert!(scores[&"ancient_op"] < 0.01); + } +} + +#[cfg(test)] +mod mincut_attention_tests { + #[test] + fn test_mincut_identifies_bottleneck() { + // Graph with clear bottleneck + // [A] [B] + // \ / + // [C] <- bottleneck + // / \ + // [D] [E] + let dag = create_bottleneck_dag(); + + let attention = MinCutGatedAttention::new(MinCutConfig::default()); + let scores = attention.forward(&dag).unwrap(); + + // Bottleneck node should have highest gating weight + assert!(scores[&"C"].gate_value > 0.9); + } + + #[test] + fn test_mincut_parallel_paths() { + // Graph with parallel paths (no single bottleneck) + let dag = create_parallel_dag(); + + let attention = MinCutGatedAttention::new(MinCutConfig::default()); + let scores = attention.forward(&dag).unwrap(); + + // All paths should have similar gating + let variance = compute_variance(&scores.values().map(|s| s.gate_value)); + assert!(variance < 0.1); + } +} +``` + +#### SONA Learning Tests + +```rust +// tests/unit/sona/mod.rs + +#[cfg(test)] +mod micro_lora_tests { + #[test] + fn test_micro_lora_rank_constraint() { + let config = MicroLoraConfig { + rank: 2, + alpha: 1.0, + }; + let lora = MicroLora::new(config, 256); + + assert_eq!(lora.a_matrix.shape(), (256, 2)); + assert_eq!(lora.b_matrix.shape(), (2, 256)); + } + + #[test] + fn test_micro_lora_adaptation_speed() { + let lora = MicroLora::new(MicroLoraConfig::default(), 256); + + let start = Instant::now(); + for _ in 0..1000 { + let gradient = random_gradient(256); + lora.adapt(&gradient); + } + let elapsed = start.elapsed(); + + // Should complete 1000 adaptations in < 100ms total + assert!(elapsed < Duration::from_millis(100)); + } + + #[test] + fn test_micro_lora_gradient_flow() { + let lora = MicroLora::new(MicroLoraConfig::default(), 256); + + // Forward pass + let input = random_vector(256); + let output = lora.forward(&input); + + // Verify output is modified + let diff: f32 = input.iter().zip(output.iter()) + .map(|(a, b)| (a - b).abs()) + .sum(); + assert!(diff > 0.0); + } +} + +#[cfg(test)] +mod ewc_tests { + #[test] + fn test_ewc_prevents_forgetting() { + let mut ewc = EwcPlusPlus::new(EwcConfig { + lambda: 5000.0, + decay: 0.99, + }); + + // Learn task A + let task_a_params = train_on_task_a(); + ewc.consolidate(&task_a_params); + + // Learn task B + let task_b_params = train_on_task_b_with_ewc(&ewc); + + // Verify task A performance is preserved + let task_a_accuracy = evaluate_task_a(&task_b_params); + assert!(task_a_accuracy > 0.8, "EWC should preserve task A performance"); + } + + #[test] + fn test_fisher_information_computation() { + let ewc = EwcPlusPlus::new(EwcConfig::default()); + + let trajectories = generate_sample_trajectories(100); + let fisher = ewc.compute_fisher(&trajectories); + + // Fisher diagonal should be non-negative + assert!(fisher.iter().all(|&f| f >= 0.0)); + + // Important parameters should have higher Fisher values + let important_idx = 0; // Assume first param is important + assert!(fisher[important_idx] > fisher.iter().sum::() / fisher.len() as f32); + } +} + +#[cfg(test)] +mod reasoning_bank_tests { + #[test] + fn test_pattern_clustering() { + let mut bank = DagReasoningBank::new(ReasoningBankConfig { + num_clusters: 10, + pattern_dim: 256, + }); + + // Add patterns from different categories + for _ in 0..100 { + bank.store_pattern(random_pattern_category_a()); + } + for _ in 0..100 { + bank.store_pattern(random_pattern_category_b()); + } + + bank.recompute_clusters(); + + // Patterns from same category should be in same cluster + let pattern_a = random_pattern_category_a(); + let pattern_b = random_pattern_category_b(); + + let cluster_a = bank.find_cluster(&pattern_a); + let cluster_a2 = bank.find_cluster(&random_pattern_category_a()); + + assert_eq!(cluster_a, cluster_a2, "Same category should cluster together"); + assert_ne!(cluster_a, bank.find_cluster(&pattern_b)); + } + + #[test] + fn test_similarity_search_accuracy() { + let mut bank = DagReasoningBank::new(ReasoningBankConfig::default()); + + // Store known patterns + let known_pattern = vec![1.0; 256]; + bank.store_pattern(DagPattern::new(known_pattern.clone(), 0.9)); + + // Query with similar pattern + let query = known_pattern.iter().map(|x| x + 0.01).collect(); + let results = bank.query_similar(&query, 1); + + assert!(!results.is_empty()); + assert!(results[0].similarity > 0.99); + } +} +``` + +### 2. Integration Tests + +#### PostgreSQL Integration + +```rust +// tests/integration/postgres/mod.rs + +#[tokio::test] +async fn test_dag_extension_lifecycle() { + let pool = create_test_pool().await; + + // Create extension + sqlx::query("CREATE EXTENSION IF NOT EXISTS ruvector_dag CASCADE") + .execute(&pool) + .await + .expect("Extension creation failed"); + + // Verify functions exist + let result: (bool,) = sqlx::query_as( + "SELECT EXISTS(SELECT 1 FROM pg_proc WHERE proname = 'dag_set_enabled')" + ) + .fetch_one(&pool) + .await + .unwrap(); + + assert!(result.0); + + // Enable DAG learning + sqlx::query("SELECT ruvector.dag_set_enabled(true)") + .execute(&pool) + .await + .expect("Enable failed"); + + // Verify enabled + let config: (bool,) = sqlx::query_as( + "SELECT enabled FROM ruvector.dag_config()" + ) + .fetch_one(&pool) + .await + .unwrap(); + + assert!(config.0); +} + +#[tokio::test] +async fn test_query_analysis_flow() { + let pool = setup_dag_extension().await; + + // Create test table with vectors + sqlx::query(r#" + CREATE TABLE IF NOT EXISTS test_vectors ( + id SERIAL PRIMARY KEY, + embedding vector(128), + category TEXT + ) + "#) + .execute(&pool) + .await + .unwrap(); + + // Insert test data + for i in 0..1000 { + sqlx::query(r#" + INSERT INTO test_vectors (embedding, category) + VALUES ($1::vector, $2) + "#) + .bind(format!("[{}]", (0..128).map(|_| rand::random::()).map(|x| x.to_string()).collect::>().join(","))) + .bind(format!("cat_{}", i % 10)) + .execute(&pool) + .await + .unwrap(); + } + + // Analyze query + let analysis: Vec = sqlx::query_as(r#" + SELECT * FROM ruvector.dag_analyze_plan($1) + "#) + .bind("SELECT * FROM test_vectors WHERE embedding <-> '[0.1, 0.2, ...]' < 0.5 LIMIT 10") + .fetch_all(&pool) + .await + .unwrap(); + + assert!(!analysis.is_empty()); + assert!(analysis.iter().any(|r| r.operator_type == "SeqScan" || r.operator_type == "HnswScan")); +} + +#[tokio::test] +async fn test_learning_trajectory_recording() { + let pool = setup_dag_extension().await; + + // Execute queries to generate trajectories + for _ in 0..10 { + sqlx::query("SELECT * FROM test_vectors ORDER BY embedding <-> $1 LIMIT 5") + .bind(random_vector_string(128)) + .execute(&pool) + .await + .unwrap(); + } + + // Wait for background learning + tokio::time::sleep(Duration::from_secs(2)).await; + + // Check trajectories were recorded + let count: (i64,) = sqlx::query_as( + "SELECT COUNT(*) FROM ruvector.dag_trajectory_history()" + ) + .fetch_one(&pool) + .await + .unwrap(); + + assert!(count.0 >= 10); +} + +#[tokio::test] +async fn test_attention_mechanism_switching() { + let pool = setup_dag_extension().await; + + let mechanisms = ["topological", "causal_cone", "critical_path", "mincut_gated"]; + + for mechanism in mechanisms { + // Set mechanism + sqlx::query("SELECT ruvector.dag_set_attention($1)") + .bind(mechanism) + .execute(&pool) + .await + .unwrap(); + + // Execute query and verify it uses correct mechanism + let result: (String,) = sqlx::query_as( + "SELECT attention_mechanism FROM ruvector.dag_config()" + ) + .fetch_one(&pool) + .await + .unwrap(); + + assert_eq!(result.0, mechanism); + + // Verify attention scores are computed + let scores: Vec = sqlx::query_as(r#" + SELECT * FROM ruvector.dag_attention_scores( + 'SELECT * FROM test_vectors LIMIT 10', + $1 + ) + "#) + .bind(mechanism) + .fetch_all(&pool) + .await + .unwrap(); + + assert!(!scores.is_empty()); + assert!(scores.iter().all(|s| s.attention_weight >= 0.0 && s.attention_weight <= 1.0)); + } +} +``` + +#### QuDAG Integration + +```rust +// tests/integration/qudag/mod.rs + +#[tokio::test] +async fn test_qudag_connection() { + // Start mock QuDAG server + let mock_server = MockQuDagServer::start().await; + + let pool = setup_dag_extension().await; + + // Connect to mock server + let result: (bool, String) = sqlx::query_as( + "SELECT connected, node_id FROM ruvector.qudag_connect($1)" + ) + .bind(&mock_server.endpoint()) + .fetch_one(&pool) + .await + .unwrap(); + + assert!(result.0); + assert!(!result.1.is_empty()); +} + +#[tokio::test] +async fn test_pattern_proposal_flow() { + let mock_server = MockQuDagServer::start().await; + let pool = setup_dag_extension().await; + + // Connect + sqlx::query("SELECT * FROM ruvector.qudag_connect($1)") + .bind(&mock_server.endpoint()) + .execute(&pool) + .await + .unwrap(); + + // Propose pattern + let result: (String, String) = sqlx::query_as(r#" + SELECT proposal_id, status + FROM ruvector.qudag_propose_pattern($1::vector, $2::jsonb, 10.0) + "#) + .bind(random_vector_string(256)) + .bind(r#"{"source": "test", "quality": 0.9}"#) + .fetch_one(&pool) + .await + .unwrap(); + + assert!(!result.0.is_empty()); + assert_eq!(result.1, "pending"); + + // Simulate consensus + mock_server.finalize_proposal(&result.0).await; + + // Check status + let status: (String, bool) = sqlx::query_as( + "SELECT status, finalized FROM ruvector.qudag_proposal_status($1)" + ) + .bind(&result.0) + .fetch_one(&pool) + .await + .unwrap(); + + assert!(status.1); +} + +#[tokio::test] +async fn test_ml_kem_encryption() { + let pool = setup_dag_extension().await; + + // Generate keypair + let keypair: (Vec, String) = sqlx::query_as( + "SELECT public_key, secret_key_id FROM ruvector.qudag_generate_kem_keypair()" + ) + .fetch_one(&pool) + .await + .unwrap(); + + // Encrypt + let plaintext = b"secret pattern data"; + let encrypted: (Vec, Vec) = sqlx::query_as( + "SELECT ciphertext, encapsulated_key FROM ruvector.qudag_encrypt($1, $2)" + ) + .bind(plaintext.as_slice()) + .bind(&keypair.0) + .fetch_one(&pool) + .await + .unwrap(); + + // Decrypt + let decrypted: (Vec,) = sqlx::query_as( + "SELECT ruvector.qudag_decrypt($1, $2, $3)" + ) + .bind(&encrypted.0) + .bind(&encrypted.1) + .bind(&keypair.1) + .fetch_one(&pool) + .await + .unwrap(); + + assert_eq!(&decrypted.0, plaintext); +} +``` + +### 3. Property-Based Tests + +```rust +// tests/property/mod.rs + +use proptest::prelude::*; + +proptest! { + #![proptest_config(ProptestConfig::with_cases(1000))] + + #[test] + fn attention_scores_sum_to_one( + nodes in prop::collection::vec(any::(), 1..100) + ) { + let dag = QueryDag::from_nodes(nodes); + let attention = TopologicalAttention::new(TopologicalConfig::default()); + + if let Ok(scores) = attention.forward(&dag) { + let sum: f32 = scores.values().sum(); + prop_assert!((sum - 1.0).abs() < 0.001, "Scores should sum to 1.0, got {}", sum); + } + } + + #[test] + fn mincut_capacity_is_positive( + edges in prop::collection::vec((0usize..100, 0usize..100, 0.0f32..100.0), 1..500) + ) { + let mut engine = DagMinCutEngine::new(); + for (u, v, w) in edges { + if u != v { + engine.add_edge(u, v, w); + } + } + + if let Ok(cut) = engine.compute_mincut(0, 99) { + prop_assert!(cut.capacity >= 0.0); + } + } + + #[test] + fn ewc_loss_increases_with_deviation( + original in prop::collection::vec(-1.0f32..1.0, 256), + deviation in 0.0f32..1.0 + ) { + let ewc = EwcPlusPlus::new(EwcConfig::default()); + ewc.consolidate(&original); + + let deviated: Vec = original.iter() + .map(|x| x + deviation) + .collect(); + + let loss_original = ewc.penalty(&original); + let loss_deviated = ewc.penalty(&deviated); + + prop_assert!( + loss_deviated >= loss_original, + "EWC loss should increase with deviation" + ); + } + + #[test] + fn pattern_similarity_is_symmetric( + pattern_a in prop::collection::vec(-1.0f32..1.0, 256), + pattern_b in prop::collection::vec(-1.0f32..1.0, 256) + ) { + let bank = DagReasoningBank::new(ReasoningBankConfig::default()); + + let sim_ab = bank.compute_similarity(&pattern_a, &pattern_b); + let sim_ba = bank.compute_similarity(&pattern_b, &pattern_a); + + prop_assert!( + (sim_ab - sim_ba).abs() < 1e-6, + "Similarity should be symmetric" + ); + } + + #[test] + fn trajectory_buffer_maintains_capacity( + trajectories in prop::collection::vec(any::(), 0..2000) + ) { + let buffer = DagTrajectoryBuffer::new(1000); + + for t in trajectories { + buffer.push(t); + } + + prop_assert!( + buffer.len() <= 1000, + "Buffer should not exceed capacity" + ); + } +} +``` + +### 4. Benchmark Tests + +```rust +// benches/dag_benchmarks.rs + +use criterion::{criterion_group, criterion_main, Criterion, BenchmarkId}; + +fn attention_benchmarks(c: &mut Criterion) { + let mut group = c.benchmark_group("attention_mechanisms"); + + for size in [10, 100, 500, 1000] { + let dag = create_random_dag(size); + + group.bench_with_input( + BenchmarkId::new("topological", size), + &dag, + |b, dag| { + let attention = TopologicalAttention::new(TopologicalConfig::default()); + b.iter(|| attention.forward(dag)) + } + ); + + group.bench_with_input( + BenchmarkId::new("causal_cone", size), + &dag, + |b, dag| { + let attention = CausalConeAttention::new(CausalConeConfig::default()); + b.iter(|| attention.forward(dag)) + } + ); + + group.bench_with_input( + BenchmarkId::new("critical_path", size), + &dag, + |b, dag| { + let attention = CriticalPathAttention::new(CriticalPathConfig::default()); + b.iter(|| attention.forward(dag)) + } + ); + + group.bench_with_input( + BenchmarkId::new("mincut_gated", size), + &dag, + |b, dag| { + let attention = MinCutGatedAttention::new(MinCutConfig::default()); + b.iter(|| attention.forward(dag)) + } + ); + } + + group.finish(); +} + +fn sona_benchmarks(c: &mut Criterion) { + let mut group = c.benchmark_group("sona_learning"); + + group.bench_function("micro_lora_adapt", |b| { + let lora = MicroLora::new(MicroLoraConfig::default(), 256); + let gradient = random_gradient(256); + b.iter(|| lora.adapt(&gradient)) + }); + + group.bench_function("ewc_penalty_256", |b| { + let ewc = EwcPlusPlus::new(EwcConfig::default()); + ewc.consolidate(&random_params(256)); + let params = random_params(256); + b.iter(|| ewc.penalty(¶ms)) + }); + + for pattern_count in [100, 1000, 10000] { + let mut bank = DagReasoningBank::new(ReasoningBankConfig::default()); + for _ in 0..pattern_count { + bank.store_pattern(random_pattern()); + } + + group.bench_with_input( + BenchmarkId::new("pattern_search", pattern_count), + &bank, + |b, bank| { + let query = random_pattern_vector(); + b.iter(|| bank.query_similar(&query, 5)) + } + ); + } + + group.finish(); +} + +fn mincut_benchmarks(c: &mut Criterion) { + let mut group = c.benchmark_group("mincut_operations"); + + for nodes in [100, 500, 1000, 5000] { + let graph = create_random_graph(nodes, nodes * 3); + + group.bench_with_input( + BenchmarkId::new("compute_mincut", nodes), + &graph, + |b, graph| { + let engine = DagMinCutEngine::from_graph(graph); + b.iter(|| engine.compute_mincut(0, nodes - 1)) + } + ); + + group.bench_with_input( + BenchmarkId::new("dynamic_update", nodes), + &graph, + |b, graph| { + let mut engine = DagMinCutEngine::from_graph(graph); + engine.compute_mincut(0, nodes - 1).unwrap(); + b.iter(|| engine.update_edge(rand::random::() % nodes, rand::random::() % nodes, rand::random())) + } + ); + } + + group.finish(); +} + +fn postgres_benchmarks(c: &mut Criterion) { + let rt = tokio::runtime::Runtime::new().unwrap(); + let pool = rt.block_on(setup_benchmark_pool()); + + let mut group = c.benchmark_group("postgres_operations"); + + group.bench_function("dag_analyze_plan", |b| { + b.to_async(&rt).iter(|| async { + sqlx::query("SELECT * FROM ruvector.dag_analyze_plan($1)") + .bind(BENCHMARK_QUERY) + .execute(&pool) + .await + }) + }); + + group.bench_function("dag_attention_scores", |b| { + b.to_async(&rt).iter(|| async { + sqlx::query("SELECT * FROM ruvector.dag_attention_scores($1, 'auto')") + .bind(BENCHMARK_QUERY) + .execute(&pool) + .await + }) + }); + + group.bench_function("pattern_similarity_search", |b| { + let query_vec = random_vector_string(256); + b.to_async(&rt).iter(|| async { + sqlx::query("SELECT * FROM ruvector.dag_query_patterns($1::vector, 10, 0.5)") + .bind(&query_vec) + .execute(&pool) + .await + }) + }); + + group.finish(); +} + +criterion_group!( + benches, + attention_benchmarks, + sona_benchmarks, + mincut_benchmarks, + postgres_benchmarks +); +criterion_main!(benches); +``` + +### 5. Performance Targets + +| Component | Metric | Target | Method | +|-----------|--------|--------|--------| +| TopologicalAttention | Latency (100 nodes) | < 50μs | Benchmark | +| CausalConeAttention | Latency (100 nodes) | < 100μs | Benchmark | +| CriticalPathAttention | Latency (100 nodes) | < 75μs | Benchmark | +| MinCutGatedAttention | Latency (100 nodes) | < 200μs | Benchmark | +| MicroLoRA | Adaptation | < 100μs | Benchmark | +| EWC++ | Penalty computation | < 10μs | Benchmark | +| Pattern search | 10K patterns | < 2ms | Benchmark | +| MinCut update | 5K nodes | O(n^0.12) amortized | Theoretical | +| Query analysis | End-to-end | < 5ms | Integration | +| Learning cycle | Full | < 100ms | Integration | + +### 6. Continuous Integration + +```yaml +# .github/workflows/dag-tests.yml + +name: Neural DAG Tests + +on: + push: + paths: + - 'ruvector-dag/**' + - 'ruvector-postgres/**' + pull_request: + paths: + - 'ruvector-dag/**' + - 'ruvector-postgres/**' + +jobs: + unit-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - name: Run unit tests + run: cargo test -p ruvector-dag --lib + + integration-tests: + runs-on: ubuntu-latest + services: + postgres: + image: postgres:15 + env: + POSTGRES_PASSWORD: test + ports: + - 5432:5432 + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - name: Run integration tests + run: cargo test -p ruvector-dag --test '*' + env: + DATABASE_URL: postgres://postgres:test@localhost/postgres + + property-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - name: Run property tests + run: cargo test -p ruvector-dag --test property -- --test-threads=1 + env: + PROPTEST_CASES: 10000 + + benchmarks: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - name: Run benchmarks + run: cargo bench -p ruvector-dag -- --noplot + - name: Check performance regression + run: | + cargo bench -p ruvector-dag -- --noplot --save-baseline new + cargo bench -p ruvector-dag -- --noplot --baseline main --load-baseline new + + coverage: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + components: llvm-tools-preview + - uses: taiki-e/install-action@cargo-llvm-cov + - name: Generate coverage + run: cargo llvm-cov -p ruvector-dag --lcov --output-path lcov.info + - uses: codecov/codecov-action@v3 + with: + files: lcov.info +``` + +### 7. Test Data Generation + +```rust +// tests/fixtures/mod.rs + +/// Generate realistic query DAGs for testing +pub fn generate_realistic_dag(complexity: DagComplexity) -> QueryDag { + match complexity { + DagComplexity::Simple => { + // SELECT * FROM t WHERE x = 1 + let mut dag = QueryDag::new(); + dag.add_node(0, OperatorNode::seq_scan("t")); + dag.add_node(1, OperatorNode::filter("x = 1")); + dag.add_edge(0, 1); + dag + } + DagComplexity::JoinQuery => { + // SELECT * FROM a JOIN b ON a.id = b.aid + let mut dag = QueryDag::new(); + dag.add_node(0, OperatorNode::seq_scan("a")); + dag.add_node(1, OperatorNode::seq_scan("b")); + dag.add_node(2, OperatorNode::hash_join()); + dag.add_edge(0, 2); + dag.add_edge(1, 2); + dag + } + DagComplexity::VectorSearch => { + // Vector similarity search with join + let mut dag = QueryDag::new(); + dag.add_node(0, OperatorNode::hnsw_scan("idx_vectors")); + dag.add_node(1, OperatorNode::seq_scan("metadata")); + dag.add_node(2, OperatorNode::nested_loop_join()); + dag.add_node(3, OperatorNode::sort("similarity")); + dag.add_node(4, OperatorNode::limit(100)); + dag.add_edge(0, 2); + dag.add_edge(1, 2); + dag.add_edge(2, 3); + dag.add_edge(3, 4); + dag + } + DagComplexity::Complex => { + // Multi-table join with aggregation + generate_complex_dag(10, 20) + } + } +} + +/// Generate patterns that simulate learned behavior +pub fn generate_learned_patterns(count: usize) -> Vec { + (0..count) + .map(|i| { + let category = i % 5; + let base_vector = match category { + 0 => generate_scan_pattern_vector(), + 1 => generate_join_pattern_vector(), + 2 => generate_aggregate_pattern_vector(), + 3 => generate_sort_pattern_vector(), + _ => generate_mixed_pattern_vector(), + }; + + DagPattern { + vector: add_noise(&base_vector, 0.1), + quality_score: 0.7 + (rand::random::() * 0.3), + metadata: json!({ + "category": category, + "source": "synthetic", + "created": chrono::Utc::now() + }), + } + }) + .collect() +} +``` + +## Test Execution Commands + +```bash +# Run all tests +cargo test -p ruvector-dag + +# Run unit tests only +cargo test -p ruvector-dag --lib + +# Run integration tests +cargo test -p ruvector-dag --test '*' + +# Run property tests with more cases +PROPTEST_CASES=10000 cargo test -p ruvector-dag --test property + +# Run benchmarks +cargo bench -p ruvector-dag + +# Run with coverage +cargo llvm-cov -p ruvector-dag + +# Run specific test +cargo test -p ruvector-dag test_topological_attention_decay + +# Run tests with logging +RUST_LOG=debug cargo test -p ruvector-dag -- --nocapture +``` + +--- + +*Document: 10-TESTING-STRATEGY.md | Version: 1.0 | Last Updated: 2025-01-XX* diff --git a/docs/dag/11-AGENT-TASKS.md b/docs/dag/11-AGENT-TASKS.md new file mode 100644 index 000000000..03dda937a --- /dev/null +++ b/docs/dag/11-AGENT-TASKS.md @@ -0,0 +1,881 @@ +# Agent Task Assignments + +## Overview + +Task breakdown for 15-agent swarm implementing the Neural DAG Learning system. Each agent has specific responsibilities, dependencies, and deliverables. + +## Swarm Topology + +``` + ┌─────────────────────┐ + │ QUEEN AGENT │ + │ (Coordinator) │ + │ Agent #0 │ + └──────────┬──────────┘ + │ + ┌──────────────────────┼──────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌───────────────┐ ┌───────────────┐ +│ CORE TEAM │ │ POSTGRES TEAM │ │ QUDAG TEAM │ +│ Agents 1-5 │ │ Agents 6-9 │ │ Agents 10-12 │ +└───────────────┘ └───────────────┘ └───────────────┘ + │ + ┌──────────┴──────────┐ + ▼ ▼ + ┌───────────────┐ ┌───────────────┐ + │ TESTING TEAM │ │ DOCS TEAM │ + │ Agents 13-14 │ │ Agent 15 │ + └───────────────┘ └───────────────┘ +``` + +## Agent Assignments + +--- + +### Agent #0: Queen Coordinator + +**Type**: `queen-coordinator` + +**Role**: Central orchestration, dependency management, conflict resolution + +**Responsibilities**: +- Monitor all agent progress via memory coordination +- Resolve cross-team dependencies and conflicts +- Manage swarm-wide configuration +- Aggregate status reports +- Make strategic decisions on implementation order +- Coordinate code reviews between teams + +**Deliverables**: +- Swarm coordination logs +- Dependency resolution decisions +- Final integration verification + +**Memory Keys**: +- `swarm/queen/status` - Overall swarm status +- `swarm/queen/decisions` - Strategic decisions log +- `swarm/queen/dependencies` - Cross-agent dependency tracking + +**No direct code output** - Coordination only + +--- + +### Agent #1: Core DAG Engine + +**Type**: `coder` + +**Role**: Core DAG data structures and algorithms + +**Responsibilities**: +1. Implement `QueryDag` structure +2. Implement `OperatorNode` and `OperatorType` +3. Implement DAG traversal algorithms (topological sort, DFS, BFS) +4. Implement edge/node management +5. Implement DAG serialization/deserialization + +**Files to Create/Modify**: +``` +ruvector-dag/src/ +├── lib.rs +├── dag/ +│ ├── mod.rs +│ ├── query_dag.rs +│ ├── operator_node.rs +│ ├── traversal.rs +│ └── serialization.rs +``` + +**Dependencies**: None (foundational) + +**Blocked By**: None + +**Blocks**: Agents 2, 3, 4, 6 + +**Deliverables**: +- [ ] `QueryDag` struct with node/edge management +- [ ] `OperatorNode` with all operator types +- [ ] Topological sort implementation +- [ ] Cycle detection +- [ ] JSON/binary serialization + +**Estimated Complexity**: Medium + +--- + +### Agent #2: Attention Mechanisms (Basic) + +**Type**: `coder` + +**Role**: Implement first 4 attention mechanisms + +**Responsibilities**: +1. Implement `DagAttention` trait +2. Implement `TopologicalAttention` +3. Implement `CausalConeAttention` +4. Implement `CriticalPathAttention` +5. Implement `MinCutGatedAttention` + +**Files to Create/Modify**: +``` +ruvector-dag/src/attention/ +├── mod.rs +├── traits.rs +├── topological.rs +├── causal_cone.rs +├── critical_path.rs +└── mincut_gated.rs +``` + +**Dependencies**: Agent #1 (QueryDag) + +**Blocked By**: Agent #1 + +**Blocks**: Agents 6, 13 + +**Deliverables**: +- [ ] `DagAttention` trait definition +- [ ] `TopologicalAttention` with decay +- [ ] `CausalConeAttention` with temporal awareness +- [ ] `CriticalPathAttention` with path computation +- [ ] `MinCutGatedAttention` with flow-based gating + +**Estimated Complexity**: High + +--- + +### Agent #3: Attention Mechanisms (Advanced) + +**Type**: `coder` + +**Role**: Implement advanced attention mechanisms + +**Responsibilities**: +1. Implement `HierarchicalLorentzAttention` +2. Implement `ParallelBranchAttention` +3. Implement `TemporalBTSPAttention` +4. Implement `AttentionSelector` (UCB bandit) +5. Implement attention caching + +**Files to Create/Modify**: +``` +ruvector-dag/src/attention/ +├── hierarchical_lorentz.rs +├── parallel_branch.rs +├── temporal_btsp.rs +├── selector.rs +└── cache.rs +``` + +**Dependencies**: Agent #1 (QueryDag), Agent #2 (DagAttention trait) + +**Blocked By**: Agents #1, #2 + +**Blocks**: Agents 6, 13 + +**Deliverables**: +- [ ] `HierarchicalLorentzAttention` with hyperbolic ops +- [ ] `ParallelBranchAttention` with branch detection +- [ ] `TemporalBTSPAttention` with eligibility traces +- [ ] `AttentionSelector` with UCB selection +- [ ] LRU attention cache + +**Estimated Complexity**: Very High + +--- + +### Agent #4: SONA Integration + +**Type**: `coder` + +**Role**: Self-Optimizing Neural Architecture integration + +**Responsibilities**: +1. Implement `DagSonaEngine` +2. Implement `MicroLoRA` adaptation +3. Implement `DagTrajectoryBuffer` +4. Implement `DagReasoningBank` +5. Implement `EwcPlusPlus` constraints + +**Files to Create/Modify**: +``` +ruvector-dag/src/sona/ +├── mod.rs +├── engine.rs +├── micro_lora.rs +├── trajectory.rs +├── reasoning_bank.rs +└── ewc.rs +``` + +**Dependencies**: Agent #1 (QueryDag) + +**Blocked By**: Agent #1 + +**Blocks**: Agents 6, 7, 13 + +**Deliverables**: +- [ ] `DagSonaEngine` orchestration +- [ ] `MicroLoRA` rank-2 adaptation (<100μs) +- [ ] Lock-free trajectory buffer +- [ ] K-means++ clustering for patterns +- [ ] EWC++ with Fisher information + +**Estimated Complexity**: Very High + +--- + +### Agent #5: MinCut Optimization + +**Type**: `coder` + +**Role**: Subpolynomial min-cut algorithms + +**Responsibilities**: +1. Implement `DagMinCutEngine` +2. Implement `LocalKCut` oracle +3. Implement dynamic update algorithms +4. Implement bottleneck detection +5. Implement redundancy suggestions + +**Files to Create/Modify**: +``` +ruvector-dag/src/mincut/ +├── mod.rs +├── engine.rs +├── local_kcut.rs +├── dynamic_updates.rs +├── bottleneck.rs +└── redundancy.rs +``` + +**Dependencies**: Agent #1 (QueryDag) + +**Blocked By**: Agent #1 + +**Blocks**: Agent #2 (MinCutGatedAttention), Agent #6 + +**Deliverables**: +- [ ] `DagMinCutEngine` with O(n^0.12) updates +- [ ] `LocalKCut` oracle implementation +- [ ] Hierarchical decomposition +- [ ] Bottleneck scoring algorithm +- [ ] Redundancy recommendation engine + +**Estimated Complexity**: Very High + +--- + +### Agent #6: PostgreSQL Core Integration + +**Type**: `backend-dev` + +**Role**: Core PostgreSQL extension integration + +**Responsibilities**: +1. Set up pgrx extension structure +2. Implement GUC variables +3. Implement global state management +4. Implement query hooks (planner, executor) +5. Implement background worker registration + +**Files to Create/Modify**: +``` +ruvector-postgres/src/dag/ +├── mod.rs +├── extension.rs +├── guc.rs +├── state.rs +├── hooks.rs +└── worker.rs +``` + +**Dependencies**: Agents #1-5 (all core components) + +**Blocked By**: Agents #1, #2, #3, #4, #5 + +**Blocks**: Agents #7, #8, #9 + +**Deliverables**: +- [ ] Extension scaffolding with pgrx +- [ ] All GUC variables from spec +- [ ] Thread-safe global state (DashMap) +- [ ] Planner hook for DAG analysis +- [ ] Executor hooks for trajectory capture +- [ ] Background worker main loop + +**Estimated Complexity**: High + +--- + +### Agent #7: PostgreSQL SQL Functions (Part 1) + +**Type**: `backend-dev` + +**Role**: Core SQL function implementations + +**Responsibilities**: +1. Configuration functions +2. Query analysis functions +3. Attention functions +4. Basic status/health functions + +**Files to Create/Modify**: +``` +ruvector-postgres/src/dag/ +├── functions/ +│ ├── mod.rs +│ ├── config.rs +│ ├── analysis.rs +│ ├── attention.rs +│ └── status.rs +``` + +**SQL Functions**: +- `dag_set_enabled` +- `dag_set_learning_rate` +- `dag_set_attention` +- `dag_configure_sona` +- `dag_config` +- `dag_status` +- `dag_analyze_plan` +- `dag_critical_path` +- `dag_bottlenecks` +- `dag_attention_scores` +- `dag_attention_matrix` + +**Dependencies**: Agent #6 (PostgreSQL core) + +**Blocked By**: Agent #6 + +**Blocks**: Agent #13 + +**Deliverables**: +- [ ] All configuration SQL functions +- [ ] Query analysis functions +- [ ] Attention computation functions +- [ ] Status reporting functions + +**Estimated Complexity**: Medium + +--- + +### Agent #8: PostgreSQL SQL Functions (Part 2) + +**Type**: `backend-dev` + +**Role**: Learning and pattern SQL functions + +**Responsibilities**: +1. Pattern management functions +2. Trajectory functions +3. Learning control functions +4. Self-healing functions + +**Files to Create/Modify**: +``` +ruvector-postgres/src/dag/ +├── functions/ +│ ├── patterns.rs +│ ├── trajectories.rs +│ ├── learning.rs +│ └── healing.rs +``` + +**SQL Functions**: +- `dag_store_pattern` +- `dag_query_patterns` +- `dag_pattern_clusters` +- `dag_consolidate_patterns` +- `dag_record_trajectory` +- `dag_trajectory_history` +- `dag_learn_now` +- `dag_reset_learning` +- `dag_health_report` +- `dag_anomalies` +- `dag_auto_repair` + +**Dependencies**: Agent #6 (PostgreSQL core), Agent #4 (SONA) + +**Blocked By**: Agents #4, #6 + +**Blocks**: Agent #13 + +**Deliverables**: +- [ ] Pattern CRUD functions +- [ ] Trajectory management functions +- [ ] Learning control functions +- [ ] Health and healing functions + +**Estimated Complexity**: Medium + +--- + +### Agent #9: Self-Healing System + +**Type**: `coder` + +**Role**: Autonomous self-healing implementation + +**Responsibilities**: +1. Implement `AnomalyDetector` +2. Implement `IndexHealthChecker` +3. Implement `LearningDriftDetector` +4. Implement repair strategies +5. Implement healing orchestrator + +**Files to Create/Modify**: +``` +ruvector-dag/src/healing/ +├── mod.rs +├── anomaly.rs +├── index_health.rs +├── drift_detector.rs +├── strategies.rs +└── orchestrator.rs +``` + +**Dependencies**: Agent #4 (SONA), Agent #6 (PostgreSQL hooks) + +**Blocked By**: Agents #4, #6 + +**Blocks**: Agent #8 (healing SQL functions), Agent #13 + +**Deliverables**: +- [ ] Z-score anomaly detection +- [ ] HNSW/IVFFlat health monitoring +- [ ] Pattern drift detection +- [ ] Repair strategy implementations +- [ ] Healing loop orchestration + +**Estimated Complexity**: High + +--- + +### Agent #10: QuDAG Client + +**Type**: `coder` + +**Role**: QuDAG network client implementation + +**Responsibilities**: +1. Implement `QuDagClient` +2. Implement network communication +3. Implement pattern proposal flow +4. Implement consensus validation +5. Implement pattern synchronization + +**Files to Create/Modify**: +``` +ruvector-dag/src/qudag/ +├── mod.rs +├── client.rs +├── network.rs +├── proposal.rs +├── consensus.rs +└── sync.rs +``` + +**Dependencies**: Agent #4 (patterns to propose) + +**Blocked By**: Agent #4 + +**Blocks**: Agents #11, #12 + +**Deliverables**: +- [ ] QuDAG network client +- [ ] Async communication layer +- [ ] Pattern proposal protocol +- [ ] Consensus status tracking +- [ ] Pattern sync mechanism + +**Estimated Complexity**: High + +--- + +### Agent #11: QuDAG Cryptography + +**Type**: `security-manager` + +**Role**: Quantum-resistant cryptography + +**Responsibilities**: +1. Implement ML-KEM-768 wrapper +2. Implement ML-DSA signature wrapper +3. Implement identity management +4. Implement secure key storage +5. Implement differential privacy for patterns + +**Files to Create/Modify**: +``` +ruvector-dag/src/qudag/ +├── crypto/ +│ ├── mod.rs +│ ├── ml_kem.rs +│ ├── ml_dsa.rs +│ ├── identity.rs +│ ├── keystore.rs +│ └── differential_privacy.rs +``` + +**Dependencies**: Agent #10 (QuDAG client) + +**Blocked By**: Agent #10 + +**Blocks**: Agent #12 + +**Deliverables**: +- [ ] ML-KEM-768 encrypt/decrypt +- [ ] ML-DSA sign/verify +- [ ] Identity keypair management +- [ ] Secure keystore (zeroize) +- [ ] Laplace noise for DP + +**Estimated Complexity**: High + +--- + +### Agent #12: QuDAG Token Integration + +**Type**: `backend-dev` + +**Role**: rUv token operations + +**Responsibilities**: +1. Implement staking interface +2. Implement reward claiming +3. Implement balance tracking +4. Implement token SQL functions +5. Implement governance participation + +**Files to Create/Modify**: +``` +ruvector-dag/src/qudag/ +├── tokens/ +│ ├── mod.rs +│ ├── staking.rs +│ ├── rewards.rs +│ └── governance.rs + +ruvector-postgres/src/dag/functions/ +├── qudag.rs (SQL functions for QuDAG) +``` + +**Dependencies**: Agent #10 (QuDAG client), Agent #11 (crypto) + +**Blocked By**: Agents #10, #11 + +**Blocks**: Agent #13 + +**Deliverables**: +- [ ] Staking operations +- [ ] Reward computation +- [ ] Balance queries +- [ ] QuDAG SQL functions +- [ ] Governance voting + +**Estimated Complexity**: Medium + +--- + +### Agent #13: Test Suite Developer + +**Type**: `tester` + +**Role**: Comprehensive test implementation + +**Responsibilities**: +1. Unit tests for all modules +2. Integration tests +3. Property-based tests +4. Benchmark tests +5. CI pipeline setup + +**Files to Create/Modify**: +``` +ruvector-dag/tests/ +├── unit/ +│ ├── attention/ +│ ├── sona/ +│ ├── mincut/ +│ ├── healing/ +│ └── qudag/ +├── integration/ +│ ├── postgres/ +│ └── qudag/ +├── property/ +└── fixtures/ + +ruvector-dag/benches/ +├── attention_bench.rs +├── sona_bench.rs +└── mincut_bench.rs + +.github/workflows/ +└── dag-tests.yml +``` + +**Dependencies**: All code agents (1-12) + +**Blocked By**: Agents #1-12 (tests require implementations) + +**Blocks**: None (can test incrementally) + +**Deliverables**: +- [ ] >80% unit test coverage +- [ ] All integration tests passing +- [ ] Property tests (1000+ cases) +- [ ] Benchmarks meeting performance targets +- [ ] CI/CD pipeline + +**Estimated Complexity**: High + +--- + +### Agent #14: Test Data & Fixtures + +**Type**: `tester` + +**Role**: Test data generation and fixtures + +**Responsibilities**: +1. Generate realistic query DAGs +2. Generate synthetic patterns +3. Generate trajectory data +4. Create mock QuDAG server +5. Create test databases + +**Files to Create/Modify**: +``` +ruvector-dag/tests/ +├── fixtures/ +│ ├── dag_generator.rs +│ ├── pattern_generator.rs +│ ├── trajectory_generator.rs +│ └── mock_qudag.rs +├── data/ +│ ├── sample_dags.json +│ ├── sample_patterns.bin +│ └── sample_trajectories.json +``` + +**Dependencies**: Agent #1 (DAG structure definitions) + +**Blocked By**: Agent #1 + +**Blocks**: Agent #13 (needs fixtures) + +**Deliverables**: +- [ ] DAG generator for all complexity levels +- [ ] Pattern generator for learning tests +- [ ] Mock QuDAG server for network tests +- [ ] Sample data files +- [ ] Test database setup scripts + +**Estimated Complexity**: Medium + +--- + +### Agent #15: Documentation & Examples + +**Type**: `api-docs` + +**Role**: API documentation and usage examples + +**Responsibilities**: +1. Rust API documentation +2. SQL API documentation +3. Usage examples +4. Integration guides +5. Troubleshooting guides + +**Files to Create/Modify**: +``` +ruvector-dag/ +├── README.md +├── examples/ +│ ├── basic_usage.rs +│ ├── attention_selection.rs +│ ├── learning_workflow.rs +│ └── qudag_integration.rs + +docs/dag/ +├── USAGE.md +├── TROUBLESHOOTING.md +└── EXAMPLES.md +``` + +**Dependencies**: All code agents (1-12) + +**Blocked By**: None (can document spec first, update with impl) + +**Blocks**: None + +**Deliverables**: +- [ ] Complete rustdoc for all public APIs +- [ ] SQL function documentation +- [ ] Working code examples +- [ ] Integration guide +- [ ] Troubleshooting guide + +**Estimated Complexity**: Medium + +--- + +## Task Dependencies Graph + +``` + ┌─────┐ + │ 0 │ Queen + └──┬──┘ + │ + ┌─────────────┼─────────────┐ + │ │ │ + ┌──┴──┐ ┌──┴──┐ ┌──┴──┐ + │ 1 │ │ 14 │ │ 15 │ + └──┬──┘ └──┬──┘ └─────┘ + │ │ + ┌────┼────┬───────┤ + │ │ │ │ + ┌──┴─┐┌─┴──┐┌┴──┐┌───┴───┐ + │ 2 ││ 4 ││ 5 ││ (13) │ + └──┬─┘└─┬──┘└─┬─┘└───────┘ + │ │ │ + ┌──┴─┐ │ │ + │ 3 │ │ │ + └──┬─┘ │ │ + │ │ │ + └────┼─────┘ + │ + ┌──┴──┐ + │ 6 │ PostgreSQL Core + └──┬──┘ + │ + ┌────┼────┬────┐ + │ │ │ │ + ┌──┴─┐┌─┴──┐│ ┌──┴──┐ + │ 7 ││ 8 ││ │ 9 │ + └────┘└────┘│ └─────┘ + │ + ┌──┴──┐ + │ 10 │ QuDAG Client + └──┬──┘ + │ + ┌──┴──┐ + │ 11 │ QuDAG Crypto + └──┬──┘ + │ + ┌──┴──┐ + │ 12 │ QuDAG Tokens + └──┬──┘ + │ + ┌──┴──┐ + │ 13 │ Tests + └─────┘ +``` + +## Execution Phases + +### Phase 1: Foundation (Agents 1, 14, 15) +- Agent #1: Core DAG Engine +- Agent #14: Test fixtures (parallel) +- Agent #15: Documentation skeleton (parallel) + +**Duration**: Can start immediately +**Milestone**: QueryDag and OperatorNode complete + +### Phase 2: Core Features (Agents 2, 3, 4, 5) +- Agent #2: Basic Attention +- Agent #3: Advanced Attention (after Agent #2) +- Agent #4: SONA Integration +- Agent #5: MinCut Optimization + +**Duration**: After Phase 1 foundation +**Milestone**: All attention mechanisms and learning components + +### Phase 3: PostgreSQL Integration (Agents 6, 7, 8, 9) +- Agent #6: PostgreSQL Core +- Agent #7: SQL Functions Part 1 (after Agent #6) +- Agent #8: SQL Functions Part 2 (after Agent #6) +- Agent #9: Self-Healing (after Agent #6) + +**Duration**: After Phase 2 core features +**Milestone**: Full PostgreSQL extension functional + +### Phase 4: QuDAG Integration (Agents 10, 11, 12) +- Agent #10: QuDAG Client +- Agent #11: QuDAG Crypto (after Agent #10) +- Agent #12: QuDAG Tokens (after Agent #11) + +**Duration**: Can start after Agent #4 (SONA) +**Milestone**: Distributed pattern learning operational + +### Phase 5: Testing & Validation (Agent 13) +- Agent #13: Full test suite +- Integration testing +- Performance validation + +**Duration**: Ongoing throughout, intensive at end +**Milestone**: All tests passing, benchmarks met + +## Coordination Protocol + +### Memory Keys for Cross-Agent Communication + +``` +swarm/dag/ +├── status/ +│ ├── agent_{N}_status # Individual agent status +│ ├── phase_status # Current phase +│ └── blockers # Active blockers +├── artifacts/ +│ ├── agent_{N}_files # Files created/modified +│ ├── interfaces # Shared interface definitions +│ └── schemas # Data schemas +├── decisions/ +│ ├── api_decisions # API design decisions +│ ├── implementation # Implementation choices +│ └── conflicts # Resolved conflicts +└── metrics/ + ├── progress # Completion percentages + ├── performance # Performance measurements + └── issues # Known issues +``` + +### Communication Hooks + +Each agent MUST run before work: +```bash +npx claude-flow@alpha hooks pre-task --description "Agent #{N}: {task}" +npx claude-flow@alpha hooks session-restore --session-id "swarm-dag" +``` + +Each agent MUST run after work: +```bash +npx claude-flow@alpha hooks post-edit --file "{file}" --memory-key "swarm/dag/artifacts/agent_{N}_files" +npx claude-flow@alpha hooks post-task --task-id "agent_{N}_{task}" +``` + +## Success Criteria + +| Agent | Must Complete | Performance Target | +|-------|---------------|-------------------| +| #1 | QueryDag, traversals, serialization | - | +| #2 | 4 attention mechanisms | <100μs per mechanism | +| #3 | 3 attention mechanisms + selector | <200μs per mechanism | +| #4 | SONA engine, MicroLoRA, ReasoningBank | <100μs adaptation | +| #5 | MinCut engine, dynamic updates | O(n^0.12) amortized | +| #6 | Extension scaffold, hooks, worker | - | +| #7 | 11 SQL functions | <5ms per function | +| #8 | 11 SQL functions | <5ms per function | +| #9 | Healing system | <1s detection latency | +| #10 | QuDAG client, sync | <500ms network ops | +| #11 | ML-KEM, ML-DSA | <10ms crypto ops | +| #12 | Token operations | <100ms token ops | +| #13 | >80% coverage, all benchmarks | - | +| #14 | All fixtures, mock server | - | +| #15 | Complete docs, examples | - | + +--- + +*Document: 11-AGENT-TASKS.md | Version: 1.0 | Last Updated: 2025-01-XX* diff --git a/docs/dag/12-MILESTONES.md b/docs/dag/12-MILESTONES.md new file mode 100644 index 000000000..277d2c844 --- /dev/null +++ b/docs/dag/12-MILESTONES.md @@ -0,0 +1,757 @@ +# Implementation Milestones + +## Overview + +Structured milestone plan for implementing the Neural DAG Learning system with 15-agent swarm coordination. + +## Milestone Summary + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ NEURAL DAG LEARNING IMPLEMENTATION │ +├─────────────────────────────────────────────────────────────────────────┤ +│ M1: Foundation ████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 15% │ +│ M2: Core Attention ████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 25% │ +│ M3: SONA Learning ████████████████░░░░░░░░░░░░░░░░░░░░░░░░ 35% │ +│ M4: PostgreSQL ████████████████████████░░░░░░░░░░░░░░░░ 55% │ +│ M5: Self-Healing ████████████████████████████░░░░░░░░░░░░ 65% │ +│ M6: QuDAG Integration ████████████████████████████████░░░░░░░░ 80% │ +│ M7: Testing ████████████████████████████████████░░░░ 90% │ +│ M8: Production Ready ████████████████████████████████████████ 100% │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Milestone 1: Foundation + +**Status**: Not Started +**Priority**: Critical +**Agents**: #1, #14, #15 + +### Objectives + +- [ ] Establish core DAG data structures +- [ ] Create test fixture infrastructure +- [ ] Initialize documentation structure + +### Deliverables + +| Deliverable | Agent | Status | Notes | +|-------------|-------|--------|-------| +| `QueryDag` struct | #1 | Pending | Node/edge management | +| `OperatorNode` enum | #1 | Pending | All 15+ operator types | +| Topological sort | #1 | Pending | O(V+E) implementation | +| Cycle detection | #1 | Pending | For validation | +| DAG serialization | #1 | Pending | JSON + binary formats | +| Test DAG generator | #14 | Pending | All complexity levels | +| Mock fixtures | #14 | Pending | Sample data | +| Doc skeleton | #15 | Pending | README + guides | + +### Acceptance Criteria + +```rust +// Core functionality must work +let mut dag = QueryDag::new(); +dag.add_node(0, OperatorNode::SeqScan { table: "users".into() }); +dag.add_node(1, OperatorNode::Filter { predicate: "id > 0".into() }); +dag.add_edge(0, 1).unwrap(); + +let sorted = dag.topological_sort().unwrap(); +assert_eq!(sorted, vec![0, 1]); + +let json = dag.to_json().unwrap(); +let restored = QueryDag::from_json(&json).unwrap(); +assert_eq!(dag, restored); +``` + +### Files Created + +``` +ruvector-dag/ +├── Cargo.toml +├── src/ +│ ├── lib.rs +│ └── dag/ +│ ├── mod.rs +│ ├── query_dag.rs +│ ├── operator_node.rs +│ ├── traversal.rs +│ └── serialization.rs +└── tests/ + └── fixtures/ + ├── dag_generator.rs + └── sample_dags.json +``` + +### Exit Criteria + +- [ ] All unit tests pass for DAG module +- [ ] Benchmark: create 1000-node DAG in <10ms +- [ ] Documentation: rustdoc for all public items +- [ ] Code review approved by Queen agent + +--- + +## Milestone 2: Core Attention Mechanisms + +**Status**: Not Started +**Priority**: Critical +**Agents**: #2, #3 + +### Objectives + +- [ ] Implement all 7 attention mechanisms +- [ ] Implement attention selector (UCB bandit) +- [ ] Achieve performance targets + +### Deliverables + +| Deliverable | Agent | Status | Target | +|-------------|-------|--------|--------| +| `DagAttention` trait | #2 | Pending | - | +| `TopologicalAttention` | #2 | Pending | <50μs/100 nodes | +| `CausalConeAttention` | #2 | Pending | <100μs/100 nodes | +| `CriticalPathAttention` | #2 | Pending | <75μs/100 nodes | +| `MinCutGatedAttention` | #2 | Pending | <200μs/100 nodes | +| `HierarchicalLorentzAttention` | #3 | Pending | <150μs/100 nodes | +| `ParallelBranchAttention` | #3 | Pending | <100μs/100 nodes | +| `TemporalBTSPAttention` | #3 | Pending | <120μs/100 nodes | +| `AttentionSelector` | #3 | Pending | UCB regret O(√T) | +| Attention cache | #3 | Pending | 10K entry LRU | + +### Acceptance Criteria + +```rust +// All mechanisms implement trait +let mechanisms: Vec> = vec![ + Box::new(TopologicalAttention::new(config)), + Box::new(CausalConeAttention::new(config)), + Box::new(CriticalPathAttention::new(config)), + Box::new(MinCutGatedAttention::new(config)), + Box::new(HierarchicalLorentzAttention::new(config)), + Box::new(ParallelBranchAttention::new(config)), + Box::new(TemporalBTSPAttention::new(config)), +]; + +for mechanism in mechanisms { + let scores = mechanism.forward(&dag).unwrap(); + + // Scores sum to ~1.0 + let sum: f32 = scores.values().sum(); + assert!((sum - 1.0).abs() < 0.001); + + // All scores in [0, 1] + assert!(scores.values().all(|&s| s >= 0.0 && s <= 1.0)); +} + +// Selector chooses based on history +let mut selector = AttentionSelector::new(mechanisms.len()); +for _ in 0..100 { + let chosen = selector.select(); + let reward = simulate_query_improvement(); + selector.update(chosen, reward); +} +``` + +### Performance Benchmarks + +| Mechanism | 10 nodes | 100 nodes | 500 nodes | 1000 nodes | +|-----------|----------|-----------|-----------|------------| +| Topological | <5μs | <50μs | <200μs | <500μs | +| CausalCone | <10μs | <100μs | <400μs | <1ms | +| CriticalPath | <8μs | <75μs | <300μs | <700μs | +| MinCutGated | <20μs | <200μs | <800μs | <2ms | +| HierarchicalLorentz | <15μs | <150μs | <600μs | <1.5ms | +| ParallelBranch | <10μs | <100μs | <400μs | <1ms | +| TemporalBTSP | <12μs | <120μs | <500μs | <1.2ms | + +### Files Created + +``` +ruvector-dag/src/attention/ +├── mod.rs +├── traits.rs +├── topological.rs +├── causal_cone.rs +├── critical_path.rs +├── mincut_gated.rs +├── hierarchical_lorentz.rs +├── parallel_branch.rs +├── temporal_btsp.rs +├── selector.rs +└── cache.rs +``` + +### Exit Criteria + +- [ ] All 7 mechanisms pass unit tests +- [ ] All performance benchmarks met +- [ ] Property tests pass (1000 cases each) +- [ ] Selector converges to best mechanism in tests +- [ ] Code review approved + +--- + +## Milestone 3: SONA Learning System + +**Status**: Not Started +**Priority**: Critical +**Agents**: #4, #5 + +### Objectives + +- [ ] Implement SONA engine with two-tier learning +- [ ] Implement MinCut optimization engine +- [ ] Achieve subpolynomial update complexity + +### Deliverables + +| Deliverable | Agent | Status | Target | +|-------------|-------|--------|--------| +| `DagSonaEngine` | #4 | Pending | Orchestration | +| `MicroLoRA` | #4 | Pending | <100μs adapt | +| `DagTrajectoryBuffer` | #4 | Pending | Lock-free, 1K cap | +| `DagReasoningBank` | #4 | Pending | 100 clusters, <2ms search | +| `EwcPlusPlus` | #4 | Pending | λ=5000 default | +| `DagMinCutEngine` | #5 | Pending | - | +| `LocalKCut` oracle | #5 | Pending | Local approximation | +| Dynamic updates | #5 | Pending | O(n^0.12) amortized | +| Bottleneck detection | #5 | Pending | - | + +### Acceptance Criteria + +```rust +// SONA instant loop +let mut sona = DagSonaEngine::new(config); +let dag = create_query_dag(); + +let start = Instant::now(); +let enhanced = sona.pre_query(&dag).unwrap(); +assert!(start.elapsed() < Duration::from_micros(100)); + +// Learning from trajectory +sona.post_query(&dag, &execution_metrics); + +// Verify learning happened +let patterns = sona.reasoning_bank.query_similar(&dag.embedding(), 1); +assert!(!patterns.is_empty()); + +// MinCut dynamic updates +let mut mincut = DagMinCutEngine::new(); +mincut.build_from_dag(&large_dag); + +let timings: Vec = (0..1000) + .map(|_| { + let start = Instant::now(); + mincut.update_edge(rand_u(), rand_v(), rand_weight()); + start.elapsed() + }) + .collect(); + +let amortized = timings.iter().sum::() / 1000; +// Verify subpolynomial: amortized << O(n) +``` + +### Files Created + +``` +ruvector-dag/src/ +├── sona/ +│ ├── mod.rs +│ ├── engine.rs +│ ├── micro_lora.rs +│ ├── trajectory.rs +│ ├── reasoning_bank.rs +│ └── ewc.rs +└── mincut/ + ├── mod.rs + ├── engine.rs + ├── local_kcut.rs + ├── dynamic_updates.rs + ├── bottleneck.rs + └── redundancy.rs +``` + +### Exit Criteria + +- [ ] MicroLoRA adapts in <100μs +- [ ] Pattern search in <2ms for 10K patterns +- [ ] EWC prevents catastrophic forgetting (>80% task retention) +- [ ] MinCut updates are O(n^0.12) amortized +- [ ] All tests pass + +--- + +## Milestone 4: PostgreSQL Integration + +**Status**: Not Started +**Priority**: Critical +**Agents**: #6, #7, #8 + +### Objectives + +- [ ] Create functional PostgreSQL extension +- [ ] Implement all SQL functions +- [ ] Hook into query execution pipeline + +### Deliverables + +| Deliverable | Agent | Status | Notes | +|-------------|-------|--------|-------| +| pgrx extension setup | #6 | Pending | Extension skeleton | +| GUC variables | #6 | Pending | All config vars | +| Global state | #6 | Pending | DashMap-based | +| Planner hook | #6 | Pending | DAG analysis | +| Executor hooks | #6 | Pending | Trajectory capture | +| Background worker | #6 | Pending | Learning loop | +| Config SQL funcs | #7 | Pending | 5 functions | +| Analysis SQL funcs | #7 | Pending | 6 functions | +| Attention SQL funcs | #7 | Pending | 3 functions | +| Pattern SQL funcs | #8 | Pending | 4 functions | +| Trajectory SQL funcs | #8 | Pending | 3 functions | +| Learning SQL funcs | #8 | Pending | 4 functions | + +### Acceptance Criteria + +```sql +-- Extension loads successfully +CREATE EXTENSION ruvector_dag CASCADE; + +-- Configuration works +SELECT ruvector.dag_set_enabled(true); +SELECT ruvector.dag_set_attention('auto'); + +-- Query analysis works +SELECT * FROM ruvector.dag_analyze_plan($$ + SELECT * FROM vectors + WHERE embedding <-> '[0.1,0.2,0.3]' < 0.5 + LIMIT 10 +$$); + +-- Patterns are stored +INSERT INTO test_vectors SELECT generate_series(1,1000), random_vector(128); +SELECT COUNT(*) FROM ruvector.dag_pattern_clusters(); -- Should have clusters + +-- Learning improves over time +DO $$ +DECLARE + initial_time FLOAT8; + final_time FLOAT8; +BEGIN + -- Run workload + FOR i IN 1..100 LOOP + PERFORM * FROM test_vectors ORDER BY embedding <-> random_vector(128) LIMIT 10; + END LOOP; + + -- Check improvement + SELECT avg_improvement INTO final_time FROM ruvector.dag_status(); + RAISE NOTICE 'Improvement ratio: %', final_time; +END $$; +``` + +### Files Created + +``` +ruvector-postgres/src/dag/ +├── mod.rs +├── extension.rs +├── guc.rs +├── state.rs +├── hooks.rs +├── worker.rs +└── functions/ + ├── mod.rs + ├── config.rs + ├── analysis.rs + ├── attention.rs + ├── patterns.rs + ├── trajectories.rs + └── learning.rs +``` + +### Exit Criteria + +- [ ] Extension creates without errors +- [ ] All 25+ SQL functions work +- [ ] Query hooks capture execution data +- [ ] Background worker runs learning loop +- [ ] Integration tests pass + +--- + +## Milestone 5: Self-Healing System + +**Status**: Not Started +**Priority**: High +**Agents**: #9 + +### Objectives + +- [ ] Implement autonomous anomaly detection +- [ ] Implement automatic repair strategies +- [ ] Integrate with healing SQL functions + +### Deliverables + +| Deliverable | Status | Notes | +|-------------|--------|-------| +| `AnomalyDetector` | Pending | Z-score based | +| `IndexHealthChecker` | Pending | HNSW/IVFFlat | +| `LearningDriftDetector` | Pending | Pattern quality trends | +| `RepairStrategy` trait | Pending | Strategy interface | +| `IndexRebalanceStrategy` | Pending | Rebalance indexes | +| `PatternResetStrategy` | Pending | Reset bad patterns | +| `HealingOrchestrator` | Pending | Main loop | + +### Acceptance Criteria + +```rust +// Anomaly detection +let mut detector = AnomalyDetector::new(AnomalyConfig { + z_threshold: 3.0, + window_size: 100, +}); + +// Inject anomaly +for _ in 0..99 { + detector.observe(1.0); // Normal +} +detector.observe(100.0); // Anomaly + +let anomalies = detector.detect(); +assert!(!anomalies.is_empty()); +assert!(anomalies[0].z_score > 3.0); + +// Self-healing +let orchestrator = HealingOrchestrator::new(config); +orchestrator.run_cycle().unwrap(); + +// Verify repairs applied +let health = orchestrator.health_report(); +assert!(health.overall_score > 0.8); +``` + +### Files Created + +``` +ruvector-dag/src/healing/ +├── mod.rs +├── anomaly.rs +├── index_health.rs +├── drift_detector.rs +├── strategies.rs +└── orchestrator.rs + +ruvector-postgres/src/dag/functions/ +└── healing.rs +``` + +### Exit Criteria + +- [ ] Anomalies detected within 1s +- [ ] Repairs applied automatically +- [ ] No false positives in 24h test +- [ ] SQL healing functions work +- [ ] Integration tests pass + +--- + +## Milestone 6: QuDAG Integration + +**Status**: Not Started +**Priority**: High +**Agents**: #10, #11, #12 + +### Objectives + +- [ ] Connect to QuDAG network +- [ ] Implement quantum-resistant crypto +- [ ] Enable distributed pattern learning + +### Deliverables + +| Deliverable | Agent | Status | Notes | +|-------------|-------|--------|-------| +| `QuDagClient` | #10 | Pending | Network client | +| Pattern proposal | #10 | Pending | Submit patterns | +| Pattern sync | #10 | Pending | Receive patterns | +| Consensus validation | #10 | Pending | Track votes | +| ML-KEM-768 | #11 | Pending | Encryption | +| ML-DSA | #11 | Pending | Signatures | +| Identity management | #11 | Pending | Key generation | +| Differential privacy | #11 | Pending | Pattern noise | +| Staking interface | #12 | Pending | Token staking | +| Reward claiming | #12 | Pending | Earn rUv | +| QuDAG SQL funcs | #12 | Pending | SQL interface | + +### Acceptance Criteria + +```rust +// Connect to network +let client = QuDagClient::connect("https://qudag.example.com:8443").await?; +assert!(client.is_connected()); + +// Propose pattern with DP +let pattern = PatternProposal { + vector: pattern_vector, + metadata: metadata, + noise: laplace_noise(epsilon), +}; +let proposal_id = client.propose_pattern(pattern).await?; + +// Wait for consensus +let status = client.wait_for_consensus(&proposal_id, timeout).await?; +assert!(matches!(status, ConsensusStatus::Finalized)); + +// Sync patterns +let new_patterns = client.sync_patterns(since_round).await?; +for pattern in new_patterns { + reasoning_bank.import_pattern(pattern); +} + +// Token operations +let balance = client.get_balance().await?; +client.stake(100.0).await?; +let rewards = client.claim_rewards().await?; +``` + +### Files Created + +``` +ruvector-dag/src/qudag/ +├── mod.rs +├── client.rs +├── network.rs +├── proposal.rs +├── consensus.rs +├── sync.rs +├── crypto/ +│ ├── mod.rs +│ ├── ml_kem.rs +│ ├── ml_dsa.rs +│ ├── identity.rs +│ ├── keystore.rs +│ └── differential_privacy.rs +└── tokens/ + ├── mod.rs + ├── staking.rs + ├── rewards.rs + └── governance.rs + +ruvector-postgres/src/dag/functions/ +└── qudag.rs +``` + +### Exit Criteria + +- [ ] Connect to test QuDAG network +- [ ] Pattern proposals finalize +- [ ] Pattern sync works bidirectionally +- [ ] ML-KEM/ML-DSA operations work +- [ ] Token operations succeed +- [ ] SQL functions work + +--- + +## Milestone 7: Comprehensive Testing + +**Status**: Not Started +**Priority**: High +**Agents**: #13, #14 + +### Objectives + +- [ ] Achieve >80% test coverage +- [ ] All benchmarks meet targets +- [ ] CI/CD pipeline operational + +### Deliverables + +| Category | Count | Status | +|----------|-------|--------| +| Unit tests (attention) | 50+ | Pending | +| Unit tests (sona) | 40+ | Pending | +| Unit tests (mincut) | 30+ | Pending | +| Unit tests (healing) | 25+ | Pending | +| Unit tests (qudag) | 35+ | Pending | +| Integration tests (postgres) | 20+ | Pending | +| Integration tests (qudag) | 15+ | Pending | +| Property tests | 20+ | Pending | +| Benchmarks | 15+ | Pending | + +### Performance Verification + +| Component | Target | Test | +|-----------|--------|------| +| Topological attention | <50μs/100 nodes | Benchmark | +| MicroLoRA | <100μs | Benchmark | +| Pattern search | <2ms/10K | Benchmark | +| MinCut update | O(n^0.12) | Benchmark | +| Query analysis | <5ms | Integration | +| Full learning cycle | <100ms | Integration | + +### Coverage Targets + +``` +Overall: >80% +attention/: >90% +sona/: >85% +mincut/: >85% +healing/: >80% +qudag/: >75% +functions/: >85% +``` + +### Files Created + +``` +ruvector-dag/ +├── tests/ +│ ├── unit/ +│ │ ├── attention/ +│ │ ├── sona/ +│ │ ├── mincut/ +│ │ ├── healing/ +│ │ └── qudag/ +│ ├── integration/ +│ │ ├── postgres/ +│ │ └── qudag/ +│ ├── property/ +│ └── fixtures/ +├── benches/ +│ ├── attention_bench.rs +│ ├── sona_bench.rs +│ └── mincut_bench.rs + +.github/workflows/ +├── dag-tests.yml +└── dag-benchmarks.yml +``` + +### Exit Criteria + +- [ ] Coverage >80% +- [ ] All tests pass +- [ ] All benchmarks meet targets +- [ ] CI pipeline green +- [ ] No critical issues + +--- + +## Milestone 8: Production Ready + +**Status**: Not Started +**Priority**: Critical +**Agents**: All + +### Objectives + +- [ ] Complete documentation +- [ ] Performance optimization +- [ ] Security audit +- [ ] Release preparation + +### Deliverables + +| Deliverable | Status | +|-------------|--------| +| Complete rustdoc | Pending | +| SQL API docs | Pending | +| Usage examples | Pending | +| Integration guide | Pending | +| Troubleshooting guide | Pending | +| Performance tuning guide | Pending | +| Security review | Pending | +| CHANGELOG | Pending | +| Release notes | Pending | + +### Security Checklist + +- [ ] No secret exposure +- [ ] Input validation on all SQL functions +- [ ] Safe memory handling (no leaks) +- [ ] Cryptographic review (ML-KEM/ML-DSA) +- [ ] Differential privacy parameters validated +- [ ] No SQL injection vectors +- [ ] Resource limits enforced + +### Performance Optimization + +- [ ] Profile and optimize hot paths +- [ ] Memory usage optimization +- [ ] Cache tuning +- [ ] Query plan caching +- [ ] Background worker tuning + +### Release Checklist + +- [ ] Version bump +- [ ] CHANGELOG updated +- [ ] All tests pass +- [ ] Benchmarks verified +- [ ] Documentation complete +- [ ] Examples tested +- [ ] Binary artifacts built +- [ ] crates.io ready (if applicable) + +### Exit Criteria + +- [ ] All previous milestones complete +- [ ] Documentation complete +- [ ] Security review passed +- [ ] Performance targets met +- [ ] Ready for production deployment + +--- + +## Risk Register + +| Risk | Impact | Probability | Mitigation | +|------|--------|-------------|------------| +| MinCut complexity target not achievable | High | Medium | Fall back to O(√n) algorithm | +| PostgreSQL hook instability | High | Low | Extensive testing, fallback modes | +| QuDAG network unavailable | Medium | Medium | Local-only fallback mode | +| Performance regression | Medium | Medium | Continuous benchmarking in CI | +| Memory leaks | High | Low | Valgrind/miri testing | +| Cross-agent coordination failures | Medium | Medium | Queen agent mediation | + +## Dependencies + +### External Dependencies + +| Dependency | Version | Purpose | +|------------|---------|---------| +| pgrx | ^0.11 | PostgreSQL extension | +| tokio | ^1.0 | Async runtime | +| dashmap | ^5.0 | Concurrent hashmap | +| crossbeam | ^0.8 | Lock-free structures | +| ndarray | ^0.15 | Numeric arrays | +| ml-kem | TBD | ML-KEM-768 | +| ml-dsa | TBD | ML-DSA signatures | + +### Internal Dependencies + +- `ruvector-core`: Vector operations, SONA base +- `ruvector-graph`: GNN, attention base +- `ruvector-postgres`: Extension infrastructure + +--- + +## Completion Tracking + +| Milestone | Weight | Status | Completion | +|-----------|--------|--------|------------| +| M1: Foundation | 15% | Not Started | 0% | +| M2: Core Attention | 10% | Not Started | 0% | +| M3: SONA Learning | 10% | Not Started | 0% | +| M4: PostgreSQL | 20% | Not Started | 0% | +| M5: Self-Healing | 10% | Not Started | 0% | +| M6: QuDAG | 15% | Not Started | 0% | +| M7: Testing | 10% | Not Started | 0% | +| M8: Production | 10% | Not Started | 0% | +| **TOTAL** | **100%** | - | **0%** | + +--- + +*Document: 12-MILESTONES.md | Version: 1.0 | Last Updated: 2025-01-XX*