ruvector/crates/ruvector-postgres/src/learning/trajectory.rs
rUv 0d24f43e2b feat(postgres): Add 53 SQL function definitions for all advanced modules (#46)
* feat(postgres): Add 7 advanced AI modules to ruvector-postgres

Comprehensive implementation of advanced AI capabilities:

## New Modules (23,541 lines of code)

### 1. Self-Learning / ReasoningBank (`src/learning/`)
- Trajectory tracking for query optimization
- Pattern extraction using K-means clustering
- ReasoningBank for pattern storage and matching
- Adaptive search parameter optimization

### 2. Attention Mechanisms (`src/attention/`)
- Scaled dot-product attention (core)
- Multi-head attention with parallel heads
- Flash Attention v2 (memory-efficient)
- 10 attention types with PostgresEnum support

### 3. GNN Layers (`src/gnn/`)
- Message passing framework
- GCN (Graph Convolutional Network)
- GraphSAGE with mean/max aggregation
- Configurable aggregation methods

### 4. Hyperbolic Embeddings (`src/hyperbolic/`)
- Poincaré ball model
- Lorentz hyperboloid model
- Hyperbolic distance metrics
- Möbius operations

### 5. Sparse Vectors (`src/sparse/`)
- COO format sparse vector type
- Efficient sparse-sparse distance functions
- BM25/SPLADE compatible
- Top-k pruning operations

### 6. Graph Operations & Cypher (`src/graph/`)
- Property graph storage (nodes/edges)
- BFS, DFS, Dijkstra traversal
- Cypher query parser (AST-based)
- Query executor with pattern matching

### 7. Tiny Dancer Routing (`src/routing/`)
- FastGRNN neural network
- Agent registry with capabilities
- Multi-objective routing optimization
- Cost/latency/quality balancing

## Docker Infrastructure
- Dockerfile with pgrx 0.12.6 and PostgreSQL 16
- docker-compose.yml with test runner
- Initialization SQL with test tables
- Shell scripts for dev/test/benchmark

## Feature Flags
- `learning`, `attention`, `gnn`, `hyperbolic`
- `sparse`, `graph`, `routing`
- `ai-complete` and `graph-complete` bundles

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(docker): Copy entire workspace for pgrx build

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(docker): Build standalone crate without workspace

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: Update README to enhance clarity and structure

* fix(postgres): Resolve compilation errors and Docker build issues

- Fix simsimd Option/Result type mismatch in scaled_dot.rs
- Fix f32/f64 type conversions in poincare.rs and lorentz.rs
- Fix AVX512 missing wrapper functions by using AVX2 fallback
- Fix Vec<Vec<f32>> to JsonB for pgrx pg_extern compatibility
- Fix DashMap get() to get_mut() for mutable access
- Fix router.rs dereference for best_score comparison
- Update Dockerfile to copy pre-written SQL file for pgrx
- Simplify init.sql to use correct function names
- Add postgres-cli npm package for CLI tooling

All changes tested successfully in Docker with:
- Extension loads with AVX2 SIMD support (8 floats/op)
- Distance functions verified working
- PostgreSQL 16 container runs successfully

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Add ruvLLM examples and enhanced postgres-cli

Added from claude/ruvector-lfm2-llm-01YS5Tc7i64PyYCLecT9L1dN branch:
- examples/ruvLLM: Complete LLM inference system with SIMD optimization
  - Pretraining, benchmarking, and optimization system
  - Real SIMD-optimized CPU inference engine
  - Comprehensive SOTA benchmark suite
  - Attention mechanisms, memory management, router

Enhanced postgres-cli with full ruvector-postgres integration:
- Sparse vector operations (BM25, top-k, prune, conversions)
- Hyperbolic geometry (Poincare, Lorentz, Mobius operations)
- Agent routing (Tiny Dancer system)
- Vector quantization (binary, scalar, product)
- Enhanced graph and learning commands

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(postgres-cli): Use native ruvector type instead of pgvector

- Change createVectorTable to use ruvector type (native RuVector extension)
- Add dimensions column for metadata since ruvector is variable-length
- Update index creation to use simple btree (HNSW/IVFFlat TBD)
- Tested against Docker container with ruvector extension

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(postgres): Add 53 SQL function definitions for all advanced modules

Enable all advanced PostgreSQL extension functions by adding their SQL
definitions to the extension file. This exposes all Rust #[pg_extern]
functions to PostgreSQL.

## New SQL Functions (53 total)

### Hyperbolic Geometry (8 functions)
- ruvector_poincare_distance, ruvector_lorentz_distance
- ruvector_mobius_add, ruvector_exp_map, ruvector_log_map
- ruvector_poincare_to_lorentz, ruvector_lorentz_to_poincare
- ruvector_minkowski_dot

### Sparse Vectors (14 functions)
- ruvector_sparse_create, ruvector_sparse_from_dense
- ruvector_sparse_dot, ruvector_sparse_cosine, ruvector_sparse_l2_distance
- ruvector_sparse_add, ruvector_sparse_scale, ruvector_sparse_to_dense
- ruvector_sparse_nnz, ruvector_sparse_dim
- ruvector_bm25_score, ruvector_tf_idf, ruvector_sparse_normalize
- ruvector_sparse_topk

### GNN - Graph Neural Networks (5 functions)
- ruvector_gnn_gcn_layer, ruvector_gnn_graphsage_layer
- ruvector_gnn_gat_layer, ruvector_gnn_message_pass
- ruvector_gnn_aggregate

### Routing/Agents - "Tiny Dancer" (11 functions)
- ruvector_route_query, ruvector_route_with_context
- ruvector_calculate_agent_affinity, ruvector_select_best_agent
- ruvector_multi_agent_route, ruvector_create_agent_embedding
- ruvector_get_routing_stats, ruvector_register_agent
- ruvector_update_agent_performance, ruvector_adaptive_route
- ruvector_fastgrnn_forward

### Learning/ReasoningBank (7 functions)
- ruvector_record_trajectory, ruvector_get_verdict
- ruvector_distill_memory, ruvector_adaptive_search
- ruvector_learning_feedback, ruvector_get_learning_patterns
- ruvector_optimize_search_params

### Graph/Cypher (8 functions)
- ruvector_graph_create_node, ruvector_graph_create_edge
- ruvector_graph_get_neighbors, ruvector_graph_shortest_path
- ruvector_graph_pagerank, ruvector_cypher_query
- ruvector_graph_traverse, ruvector_graph_similarity_search

## CLI Updates
- Enabled hyperbolic geometry commands in postgres-cli
- Added vector distance and normalize commands
- Enhanced client with connection pooling and retry logic

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-12-02 22:49:29 -05:00

307 lines
8.3 KiB
Rust

//! Query trajectory tracking for learning query patterns
use std::sync::RwLock;
use std::time::{Duration, SystemTime};
/// A single query trajectory record
#[derive(Debug, Clone)]
pub struct QueryTrajectory {
/// Query vector
pub query_vector: Vec<f32>,
/// Result IDs
pub result_ids: Vec<u64>,
/// Query latency in microseconds
pub latency_us: u64,
/// Search parameters used
pub ef_search: usize,
pub probes: usize,
/// Timestamp
pub timestamp: SystemTime,
/// Relevance feedback (if provided)
pub relevant_ids: Vec<u64>,
pub irrelevant_ids: Vec<u64>,
}
impl QueryTrajectory {
/// Create a new query trajectory
pub fn new(
query_vector: Vec<f32>,
result_ids: Vec<u64>,
latency_us: u64,
ef_search: usize,
probes: usize,
) -> Self {
Self {
query_vector,
result_ids,
latency_us,
ef_search,
probes,
timestamp: SystemTime::now(),
relevant_ids: Vec::new(),
irrelevant_ids: Vec::new(),
}
}
/// Add relevance feedback
pub fn add_feedback(&mut self, relevant_ids: Vec<u64>, irrelevant_ids: Vec<u64>) {
self.relevant_ids = relevant_ids;
self.irrelevant_ids = irrelevant_ids;
}
/// Calculate precision if feedback is available
pub fn precision(&self) -> Option<f64> {
if self.relevant_ids.is_empty() {
return None;
}
let relevant_retrieved = self.result_ids.iter()
.filter(|id| self.relevant_ids.contains(id))
.count();
Some(relevant_retrieved as f64 / self.result_ids.len() as f64)
}
/// Calculate recall if feedback is available
pub fn recall(&self) -> Option<f64> {
if self.relevant_ids.is_empty() {
return None;
}
let relevant_retrieved = self.result_ids.iter()
.filter(|id| self.relevant_ids.contains(id))
.count();
Some(relevant_retrieved as f64 / self.relevant_ids.len() as f64)
}
}
/// Trajectory tracker with ring buffer
pub struct TrajectoryTracker {
/// Ring buffer of trajectories
trajectories: RwLock<Vec<QueryTrajectory>>,
/// Maximum number of trajectories to keep
max_size: usize,
/// Current write position
write_pos: RwLock<usize>,
}
impl TrajectoryTracker {
/// Create a new trajectory tracker
pub fn new(max_size: usize) -> Self {
Self {
trajectories: RwLock::new(Vec::with_capacity(max_size)),
max_size,
write_pos: RwLock::new(0),
}
}
/// Record a new trajectory
pub fn record(&self, trajectory: QueryTrajectory) {
let mut trajectories = self.trajectories.write().unwrap();
let mut pos = self.write_pos.write().unwrap();
if trajectories.len() < self.max_size {
trajectories.push(trajectory);
} else {
trajectories[*pos] = trajectory;
}
*pos = (*pos + 1) % self.max_size;
}
/// Get the most recent n trajectories
pub fn get_recent(&self, n: usize) -> Vec<QueryTrajectory> {
let trajectories = self.trajectories.read().unwrap();
let count = trajectories.len().min(n);
if count == 0 {
return Vec::new();
}
let pos = *self.write_pos.read().unwrap();
let mut result = Vec::with_capacity(count);
if trajectories.len() < self.max_size {
// Not full yet, just take last n
let start = trajectories.len().saturating_sub(count);
result.extend_from_slice(&trajectories[start..]);
} else {
// Ring buffer is full, need to handle wrap-around
for i in 0..count {
let idx = (pos + self.max_size - count + i) % self.max_size;
result.push(trajectories[idx].clone());
}
}
result
}
/// Get all trajectories
pub fn get_all(&self) -> Vec<QueryTrajectory> {
self.trajectories.read().unwrap().clone()
}
/// Get trajectories within a time window
pub fn get_since(&self, duration: Duration) -> Vec<QueryTrajectory> {
let trajectories = self.trajectories.read().unwrap();
let cutoff = SystemTime::now() - duration;
trajectories.iter()
.filter(|t| t.timestamp >= cutoff)
.cloned()
.collect()
}
/// Get trajectories with feedback only
pub fn get_with_feedback(&self) -> Vec<QueryTrajectory> {
let trajectories = self.trajectories.read().unwrap();
trajectories.iter()
.filter(|t| !t.relevant_ids.is_empty())
.cloned()
.collect()
}
/// Calculate average latency
pub fn avg_latency(&self) -> Option<f64> {
let trajectories = self.trajectories.read().unwrap();
if trajectories.is_empty() {
return None;
}
let sum: u64 = trajectories.iter().map(|t| t.latency_us).sum();
Some(sum as f64 / trajectories.len() as f64)
}
/// Get statistics
pub fn stats(&self) -> TrajectoryStats {
let trajectories = self.trajectories.read().unwrap();
if trajectories.is_empty() {
return TrajectoryStats::default();
}
let total = trajectories.len();
let with_feedback = trajectories.iter().filter(|t| !t.relevant_ids.is_empty()).count();
let avg_latency = trajectories.iter().map(|t| t.latency_us).sum::<u64>() as f64 / total as f64;
let avg_precision = if with_feedback > 0 {
trajectories.iter()
.filter_map(|t| t.precision())
.sum::<f64>() / with_feedback as f64
} else {
0.0
};
let avg_recall = if with_feedback > 0 {
trajectories.iter()
.filter_map(|t| t.recall())
.sum::<f64>() / with_feedback as f64
} else {
0.0
};
TrajectoryStats {
total_trajectories: total,
trajectories_with_feedback: with_feedback,
avg_latency_us: avg_latency,
avg_precision,
avg_recall,
}
}
}
/// Trajectory statistics
#[derive(Debug, Clone, Default)]
pub struct TrajectoryStats {
pub total_trajectories: usize,
pub trajectories_with_feedback: usize,
pub avg_latency_us: f64,
pub avg_precision: f64,
pub avg_recall: f64,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_trajectory_creation() {
let traj = QueryTrajectory::new(
vec![1.0, 2.0, 3.0],
vec![1, 2, 3],
1000,
50,
10,
);
assert_eq!(traj.query_vector, vec![1.0, 2.0, 3.0]);
assert_eq!(traj.result_ids, vec![1, 2, 3]);
assert_eq!(traj.latency_us, 1000);
}
#[test]
fn test_trajectory_feedback() {
let mut traj = QueryTrajectory::new(
vec![1.0, 2.0],
vec![1, 2, 3, 4],
1000,
50,
10,
);
traj.add_feedback(vec![1, 2, 5], vec![3]);
assert_eq!(traj.precision(), Some(0.5)); // 2 out of 4 relevant
assert_eq!(traj.recall(), Some(2.0 / 3.0)); // 2 out of 3 total relevant
}
#[test]
fn test_tracker_ring_buffer() {
let tracker = TrajectoryTracker::new(3);
// Add 5 trajectories
for i in 0..5 {
tracker.record(QueryTrajectory::new(
vec![i as f32],
vec![i],
1000,
50,
10,
));
}
let all = tracker.get_all();
assert_eq!(all.len(), 3); // Ring buffer size
// Should have trajectories 2, 3, 4 (last 3)
let recent = tracker.get_recent(3);
assert_eq!(recent.len(), 3);
}
#[test]
fn test_tracker_stats() {
let tracker = TrajectoryTracker::new(10);
tracker.record(QueryTrajectory::new(
vec![1.0],
vec![1, 2],
1000,
50,
10,
));
tracker.record(QueryTrajectory::new(
vec![2.0],
vec![3, 4],
2000,
60,
15,
));
let stats = tracker.stats();
assert_eq!(stats.total_trajectories, 2);
assert_eq!(stats.avg_latency_us, 1500.0);
}
}