mirror of
https://github.com/ruvnet/RuVector.git
synced 2026-05-27 00:25:10 +00:00
* feat(postgres): Add 7 advanced AI modules to ruvector-postgres Comprehensive implementation of advanced AI capabilities: ## New Modules (23,541 lines of code) ### 1. Self-Learning / ReasoningBank (`src/learning/`) - Trajectory tracking for query optimization - Pattern extraction using K-means clustering - ReasoningBank for pattern storage and matching - Adaptive search parameter optimization ### 2. Attention Mechanisms (`src/attention/`) - Scaled dot-product attention (core) - Multi-head attention with parallel heads - Flash Attention v2 (memory-efficient) - 10 attention types with PostgresEnum support ### 3. GNN Layers (`src/gnn/`) - Message passing framework - GCN (Graph Convolutional Network) - GraphSAGE with mean/max aggregation - Configurable aggregation methods ### 4. Hyperbolic Embeddings (`src/hyperbolic/`) - Poincaré ball model - Lorentz hyperboloid model - Hyperbolic distance metrics - Möbius operations ### 5. Sparse Vectors (`src/sparse/`) - COO format sparse vector type - Efficient sparse-sparse distance functions - BM25/SPLADE compatible - Top-k pruning operations ### 6. Graph Operations & Cypher (`src/graph/`) - Property graph storage (nodes/edges) - BFS, DFS, Dijkstra traversal - Cypher query parser (AST-based) - Query executor with pattern matching ### 7. Tiny Dancer Routing (`src/routing/`) - FastGRNN neural network - Agent registry with capabilities - Multi-objective routing optimization - Cost/latency/quality balancing ## Docker Infrastructure - Dockerfile with pgrx 0.12.6 and PostgreSQL 16 - docker-compose.yml with test runner - Initialization SQL with test tables - Shell scripts for dev/test/benchmark ## Feature Flags - `learning`, `attention`, `gnn`, `hyperbolic` - `sparse`, `graph`, `routing` - `ai-complete` and `graph-complete` bundles 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix(docker): Copy entire workspace for pgrx build 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix(docker): Build standalone crate without workspace 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: Update README to enhance clarity and structure * fix(postgres): Resolve compilation errors and Docker build issues - Fix simsimd Option/Result type mismatch in scaled_dot.rs - Fix f32/f64 type conversions in poincare.rs and lorentz.rs - Fix AVX512 missing wrapper functions by using AVX2 fallback - Fix Vec<Vec<f32>> to JsonB for pgrx pg_extern compatibility - Fix DashMap get() to get_mut() for mutable access - Fix router.rs dereference for best_score comparison - Update Dockerfile to copy pre-written SQL file for pgrx - Simplify init.sql to use correct function names - Add postgres-cli npm package for CLI tooling All changes tested successfully in Docker with: - Extension loads with AVX2 SIMD support (8 floats/op) - Distance functions verified working - PostgreSQL 16 container runs successfully 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: Add ruvLLM examples and enhanced postgres-cli Added from claude/ruvector-lfm2-llm-01YS5Tc7i64PyYCLecT9L1dN branch: - examples/ruvLLM: Complete LLM inference system with SIMD optimization - Pretraining, benchmarking, and optimization system - Real SIMD-optimized CPU inference engine - Comprehensive SOTA benchmark suite - Attention mechanisms, memory management, router Enhanced postgres-cli with full ruvector-postgres integration: - Sparse vector operations (BM25, top-k, prune, conversions) - Hyperbolic geometry (Poincare, Lorentz, Mobius operations) - Agent routing (Tiny Dancer system) - Vector quantization (binary, scalar, product) - Enhanced graph and learning commands 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix(postgres-cli): Use native ruvector type instead of pgvector - Change createVectorTable to use ruvector type (native RuVector extension) - Add dimensions column for metadata since ruvector is variable-length - Update index creation to use simple btree (HNSW/IVFFlat TBD) - Tested against Docker container with ruvector extension 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat(postgres): Add 53 SQL function definitions for all advanced modules Enable all advanced PostgreSQL extension functions by adding their SQL definitions to the extension file. This exposes all Rust #[pg_extern] functions to PostgreSQL. ## New SQL Functions (53 total) ### Hyperbolic Geometry (8 functions) - ruvector_poincare_distance, ruvector_lorentz_distance - ruvector_mobius_add, ruvector_exp_map, ruvector_log_map - ruvector_poincare_to_lorentz, ruvector_lorentz_to_poincare - ruvector_minkowski_dot ### Sparse Vectors (14 functions) - ruvector_sparse_create, ruvector_sparse_from_dense - ruvector_sparse_dot, ruvector_sparse_cosine, ruvector_sparse_l2_distance - ruvector_sparse_add, ruvector_sparse_scale, ruvector_sparse_to_dense - ruvector_sparse_nnz, ruvector_sparse_dim - ruvector_bm25_score, ruvector_tf_idf, ruvector_sparse_normalize - ruvector_sparse_topk ### GNN - Graph Neural Networks (5 functions) - ruvector_gnn_gcn_layer, ruvector_gnn_graphsage_layer - ruvector_gnn_gat_layer, ruvector_gnn_message_pass - ruvector_gnn_aggregate ### Routing/Agents - "Tiny Dancer" (11 functions) - ruvector_route_query, ruvector_route_with_context - ruvector_calculate_agent_affinity, ruvector_select_best_agent - ruvector_multi_agent_route, ruvector_create_agent_embedding - ruvector_get_routing_stats, ruvector_register_agent - ruvector_update_agent_performance, ruvector_adaptive_route - ruvector_fastgrnn_forward ### Learning/ReasoningBank (7 functions) - ruvector_record_trajectory, ruvector_get_verdict - ruvector_distill_memory, ruvector_adaptive_search - ruvector_learning_feedback, ruvector_get_learning_patterns - ruvector_optimize_search_params ### Graph/Cypher (8 functions) - ruvector_graph_create_node, ruvector_graph_create_edge - ruvector_graph_get_neighbors, ruvector_graph_shortest_path - ruvector_graph_pagerank, ruvector_cypher_query - ruvector_graph_traverse, ruvector_graph_similarity_search ## CLI Updates - Enabled hyperbolic geometry commands in postgres-cli - Added vector distance and normalize commands - Enhanced client with connection pooling and retry logic 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
307 lines
8.3 KiB
Rust
307 lines
8.3 KiB
Rust
//! Query trajectory tracking for learning query patterns
|
|
|
|
use std::sync::RwLock;
|
|
use std::time::{Duration, SystemTime};
|
|
|
|
/// A single query trajectory record
|
|
#[derive(Debug, Clone)]
|
|
pub struct QueryTrajectory {
|
|
/// Query vector
|
|
pub query_vector: Vec<f32>,
|
|
/// Result IDs
|
|
pub result_ids: Vec<u64>,
|
|
/// Query latency in microseconds
|
|
pub latency_us: u64,
|
|
/// Search parameters used
|
|
pub ef_search: usize,
|
|
pub probes: usize,
|
|
/// Timestamp
|
|
pub timestamp: SystemTime,
|
|
/// Relevance feedback (if provided)
|
|
pub relevant_ids: Vec<u64>,
|
|
pub irrelevant_ids: Vec<u64>,
|
|
}
|
|
|
|
impl QueryTrajectory {
|
|
/// Create a new query trajectory
|
|
pub fn new(
|
|
query_vector: Vec<f32>,
|
|
result_ids: Vec<u64>,
|
|
latency_us: u64,
|
|
ef_search: usize,
|
|
probes: usize,
|
|
) -> Self {
|
|
Self {
|
|
query_vector,
|
|
result_ids,
|
|
latency_us,
|
|
ef_search,
|
|
probes,
|
|
timestamp: SystemTime::now(),
|
|
relevant_ids: Vec::new(),
|
|
irrelevant_ids: Vec::new(),
|
|
}
|
|
}
|
|
|
|
/// Add relevance feedback
|
|
pub fn add_feedback(&mut self, relevant_ids: Vec<u64>, irrelevant_ids: Vec<u64>) {
|
|
self.relevant_ids = relevant_ids;
|
|
self.irrelevant_ids = irrelevant_ids;
|
|
}
|
|
|
|
/// Calculate precision if feedback is available
|
|
pub fn precision(&self) -> Option<f64> {
|
|
if self.relevant_ids.is_empty() {
|
|
return None;
|
|
}
|
|
|
|
let relevant_retrieved = self.result_ids.iter()
|
|
.filter(|id| self.relevant_ids.contains(id))
|
|
.count();
|
|
|
|
Some(relevant_retrieved as f64 / self.result_ids.len() as f64)
|
|
}
|
|
|
|
/// Calculate recall if feedback is available
|
|
pub fn recall(&self) -> Option<f64> {
|
|
if self.relevant_ids.is_empty() {
|
|
return None;
|
|
}
|
|
|
|
let relevant_retrieved = self.result_ids.iter()
|
|
.filter(|id| self.relevant_ids.contains(id))
|
|
.count();
|
|
|
|
Some(relevant_retrieved as f64 / self.relevant_ids.len() as f64)
|
|
}
|
|
}
|
|
|
|
/// Trajectory tracker with ring buffer
|
|
pub struct TrajectoryTracker {
|
|
/// Ring buffer of trajectories
|
|
trajectories: RwLock<Vec<QueryTrajectory>>,
|
|
/// Maximum number of trajectories to keep
|
|
max_size: usize,
|
|
/// Current write position
|
|
write_pos: RwLock<usize>,
|
|
}
|
|
|
|
impl TrajectoryTracker {
|
|
/// Create a new trajectory tracker
|
|
pub fn new(max_size: usize) -> Self {
|
|
Self {
|
|
trajectories: RwLock::new(Vec::with_capacity(max_size)),
|
|
max_size,
|
|
write_pos: RwLock::new(0),
|
|
}
|
|
}
|
|
|
|
/// Record a new trajectory
|
|
pub fn record(&self, trajectory: QueryTrajectory) {
|
|
let mut trajectories = self.trajectories.write().unwrap();
|
|
let mut pos = self.write_pos.write().unwrap();
|
|
|
|
if trajectories.len() < self.max_size {
|
|
trajectories.push(trajectory);
|
|
} else {
|
|
trajectories[*pos] = trajectory;
|
|
}
|
|
|
|
*pos = (*pos + 1) % self.max_size;
|
|
}
|
|
|
|
/// Get the most recent n trajectories
|
|
pub fn get_recent(&self, n: usize) -> Vec<QueryTrajectory> {
|
|
let trajectories = self.trajectories.read().unwrap();
|
|
let count = trajectories.len().min(n);
|
|
|
|
if count == 0 {
|
|
return Vec::new();
|
|
}
|
|
|
|
let pos = *self.write_pos.read().unwrap();
|
|
let mut result = Vec::with_capacity(count);
|
|
|
|
if trajectories.len() < self.max_size {
|
|
// Not full yet, just take last n
|
|
let start = trajectories.len().saturating_sub(count);
|
|
result.extend_from_slice(&trajectories[start..]);
|
|
} else {
|
|
// Ring buffer is full, need to handle wrap-around
|
|
for i in 0..count {
|
|
let idx = (pos + self.max_size - count + i) % self.max_size;
|
|
result.push(trajectories[idx].clone());
|
|
}
|
|
}
|
|
|
|
result
|
|
}
|
|
|
|
/// Get all trajectories
|
|
pub fn get_all(&self) -> Vec<QueryTrajectory> {
|
|
self.trajectories.read().unwrap().clone()
|
|
}
|
|
|
|
/// Get trajectories within a time window
|
|
pub fn get_since(&self, duration: Duration) -> Vec<QueryTrajectory> {
|
|
let trajectories = self.trajectories.read().unwrap();
|
|
let cutoff = SystemTime::now() - duration;
|
|
|
|
trajectories.iter()
|
|
.filter(|t| t.timestamp >= cutoff)
|
|
.cloned()
|
|
.collect()
|
|
}
|
|
|
|
/// Get trajectories with feedback only
|
|
pub fn get_with_feedback(&self) -> Vec<QueryTrajectory> {
|
|
let trajectories = self.trajectories.read().unwrap();
|
|
trajectories.iter()
|
|
.filter(|t| !t.relevant_ids.is_empty())
|
|
.cloned()
|
|
.collect()
|
|
}
|
|
|
|
/// Calculate average latency
|
|
pub fn avg_latency(&self) -> Option<f64> {
|
|
let trajectories = self.trajectories.read().unwrap();
|
|
if trajectories.is_empty() {
|
|
return None;
|
|
}
|
|
|
|
let sum: u64 = trajectories.iter().map(|t| t.latency_us).sum();
|
|
Some(sum as f64 / trajectories.len() as f64)
|
|
}
|
|
|
|
/// Get statistics
|
|
pub fn stats(&self) -> TrajectoryStats {
|
|
let trajectories = self.trajectories.read().unwrap();
|
|
|
|
if trajectories.is_empty() {
|
|
return TrajectoryStats::default();
|
|
}
|
|
|
|
let total = trajectories.len();
|
|
let with_feedback = trajectories.iter().filter(|t| !t.relevant_ids.is_empty()).count();
|
|
|
|
let avg_latency = trajectories.iter().map(|t| t.latency_us).sum::<u64>() as f64 / total as f64;
|
|
|
|
let avg_precision = if with_feedback > 0 {
|
|
trajectories.iter()
|
|
.filter_map(|t| t.precision())
|
|
.sum::<f64>() / with_feedback as f64
|
|
} else {
|
|
0.0
|
|
};
|
|
|
|
let avg_recall = if with_feedback > 0 {
|
|
trajectories.iter()
|
|
.filter_map(|t| t.recall())
|
|
.sum::<f64>() / with_feedback as f64
|
|
} else {
|
|
0.0
|
|
};
|
|
|
|
TrajectoryStats {
|
|
total_trajectories: total,
|
|
trajectories_with_feedback: with_feedback,
|
|
avg_latency_us: avg_latency,
|
|
avg_precision,
|
|
avg_recall,
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Trajectory statistics
|
|
#[derive(Debug, Clone, Default)]
|
|
pub struct TrajectoryStats {
|
|
pub total_trajectories: usize,
|
|
pub trajectories_with_feedback: usize,
|
|
pub avg_latency_us: f64,
|
|
pub avg_precision: f64,
|
|
pub avg_recall: f64,
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn test_trajectory_creation() {
|
|
let traj = QueryTrajectory::new(
|
|
vec![1.0, 2.0, 3.0],
|
|
vec![1, 2, 3],
|
|
1000,
|
|
50,
|
|
10,
|
|
);
|
|
|
|
assert_eq!(traj.query_vector, vec![1.0, 2.0, 3.0]);
|
|
assert_eq!(traj.result_ids, vec![1, 2, 3]);
|
|
assert_eq!(traj.latency_us, 1000);
|
|
}
|
|
|
|
#[test]
|
|
fn test_trajectory_feedback() {
|
|
let mut traj = QueryTrajectory::new(
|
|
vec![1.0, 2.0],
|
|
vec![1, 2, 3, 4],
|
|
1000,
|
|
50,
|
|
10,
|
|
);
|
|
|
|
traj.add_feedback(vec![1, 2, 5], vec![3]);
|
|
|
|
assert_eq!(traj.precision(), Some(0.5)); // 2 out of 4 relevant
|
|
assert_eq!(traj.recall(), Some(2.0 / 3.0)); // 2 out of 3 total relevant
|
|
}
|
|
|
|
#[test]
|
|
fn test_tracker_ring_buffer() {
|
|
let tracker = TrajectoryTracker::new(3);
|
|
|
|
// Add 5 trajectories
|
|
for i in 0..5 {
|
|
tracker.record(QueryTrajectory::new(
|
|
vec![i as f32],
|
|
vec![i],
|
|
1000,
|
|
50,
|
|
10,
|
|
));
|
|
}
|
|
|
|
let all = tracker.get_all();
|
|
assert_eq!(all.len(), 3); // Ring buffer size
|
|
|
|
// Should have trajectories 2, 3, 4 (last 3)
|
|
let recent = tracker.get_recent(3);
|
|
assert_eq!(recent.len(), 3);
|
|
}
|
|
|
|
#[test]
|
|
fn test_tracker_stats() {
|
|
let tracker = TrajectoryTracker::new(10);
|
|
|
|
tracker.record(QueryTrajectory::new(
|
|
vec![1.0],
|
|
vec![1, 2],
|
|
1000,
|
|
50,
|
|
10,
|
|
));
|
|
|
|
tracker.record(QueryTrajectory::new(
|
|
vec![2.0],
|
|
vec![3, 4],
|
|
2000,
|
|
60,
|
|
15,
|
|
));
|
|
|
|
let stats = tracker.stats();
|
|
assert_eq!(stats.total_trajectories, 2);
|
|
assert_eq!(stats.avg_latency_us, 1500.0);
|
|
}
|
|
}
|