ruvector/crates/ruvector-postgres/docker/README.md
rUv 073ce73612
feat(postgres): Add 53 SQL function definitions for all advanced modules (#46)
* feat(postgres): Add 7 advanced AI modules to ruvector-postgres

Comprehensive implementation of advanced AI capabilities:

## New Modules (23,541 lines of code)

### 1. Self-Learning / ReasoningBank (`src/learning/`)
- Trajectory tracking for query optimization
- Pattern extraction using K-means clustering
- ReasoningBank for pattern storage and matching
- Adaptive search parameter optimization

### 2. Attention Mechanisms (`src/attention/`)
- Scaled dot-product attention (core)
- Multi-head attention with parallel heads
- Flash Attention v2 (memory-efficient)
- 10 attention types with PostgresEnum support

### 3. GNN Layers (`src/gnn/`)
- Message passing framework
- GCN (Graph Convolutional Network)
- GraphSAGE with mean/max aggregation
- Configurable aggregation methods

### 4. Hyperbolic Embeddings (`src/hyperbolic/`)
- Poincaré ball model
- Lorentz hyperboloid model
- Hyperbolic distance metrics
- Möbius operations

### 5. Sparse Vectors (`src/sparse/`)
- COO format sparse vector type
- Efficient sparse-sparse distance functions
- BM25/SPLADE compatible
- Top-k pruning operations

### 6. Graph Operations & Cypher (`src/graph/`)
- Property graph storage (nodes/edges)
- BFS, DFS, Dijkstra traversal
- Cypher query parser (AST-based)
- Query executor with pattern matching

### 7. Tiny Dancer Routing (`src/routing/`)
- FastGRNN neural network
- Agent registry with capabilities
- Multi-objective routing optimization
- Cost/latency/quality balancing

## Docker Infrastructure
- Dockerfile with pgrx 0.12.6 and PostgreSQL 16
- docker-compose.yml with test runner
- Initialization SQL with test tables
- Shell scripts for dev/test/benchmark

## Feature Flags
- `learning`, `attention`, `gnn`, `hyperbolic`
- `sparse`, `graph`, `routing`
- `ai-complete` and `graph-complete` bundles

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(docker): Copy entire workspace for pgrx build

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(docker): Build standalone crate without workspace

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: Update README to enhance clarity and structure

* fix(postgres): Resolve compilation errors and Docker build issues

- Fix simsimd Option/Result type mismatch in scaled_dot.rs
- Fix f32/f64 type conversions in poincare.rs and lorentz.rs
- Fix AVX512 missing wrapper functions by using AVX2 fallback
- Fix Vec<Vec<f32>> to JsonB for pgrx pg_extern compatibility
- Fix DashMap get() to get_mut() for mutable access
- Fix router.rs dereference for best_score comparison
- Update Dockerfile to copy pre-written SQL file for pgrx
- Simplify init.sql to use correct function names
- Add postgres-cli npm package for CLI tooling

All changes tested successfully in Docker with:
- Extension loads with AVX2 SIMD support (8 floats/op)
- Distance functions verified working
- PostgreSQL 16 container runs successfully

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Add ruvLLM examples and enhanced postgres-cli

Added from claude/ruvector-lfm2-llm-01YS5Tc7i64PyYCLecT9L1dN branch:
- examples/ruvLLM: Complete LLM inference system with SIMD optimization
  - Pretraining, benchmarking, and optimization system
  - Real SIMD-optimized CPU inference engine
  - Comprehensive SOTA benchmark suite
  - Attention mechanisms, memory management, router

Enhanced postgres-cli with full ruvector-postgres integration:
- Sparse vector operations (BM25, top-k, prune, conversions)
- Hyperbolic geometry (Poincare, Lorentz, Mobius operations)
- Agent routing (Tiny Dancer system)
- Vector quantization (binary, scalar, product)
- Enhanced graph and learning commands

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(postgres-cli): Use native ruvector type instead of pgvector

- Change createVectorTable to use ruvector type (native RuVector extension)
- Add dimensions column for metadata since ruvector is variable-length
- Update index creation to use simple btree (HNSW/IVFFlat TBD)
- Tested against Docker container with ruvector extension

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(postgres): Add 53 SQL function definitions for all advanced modules

Enable all advanced PostgreSQL extension functions by adding their SQL
definitions to the extension file. This exposes all Rust #[pg_extern]
functions to PostgreSQL.

## New SQL Functions (53 total)

### Hyperbolic Geometry (8 functions)
- ruvector_poincare_distance, ruvector_lorentz_distance
- ruvector_mobius_add, ruvector_exp_map, ruvector_log_map
- ruvector_poincare_to_lorentz, ruvector_lorentz_to_poincare
- ruvector_minkowski_dot

### Sparse Vectors (14 functions)
- ruvector_sparse_create, ruvector_sparse_from_dense
- ruvector_sparse_dot, ruvector_sparse_cosine, ruvector_sparse_l2_distance
- ruvector_sparse_add, ruvector_sparse_scale, ruvector_sparse_to_dense
- ruvector_sparse_nnz, ruvector_sparse_dim
- ruvector_bm25_score, ruvector_tf_idf, ruvector_sparse_normalize
- ruvector_sparse_topk

### GNN - Graph Neural Networks (5 functions)
- ruvector_gnn_gcn_layer, ruvector_gnn_graphsage_layer
- ruvector_gnn_gat_layer, ruvector_gnn_message_pass
- ruvector_gnn_aggregate

### Routing/Agents - "Tiny Dancer" (11 functions)
- ruvector_route_query, ruvector_route_with_context
- ruvector_calculate_agent_affinity, ruvector_select_best_agent
- ruvector_multi_agent_route, ruvector_create_agent_embedding
- ruvector_get_routing_stats, ruvector_register_agent
- ruvector_update_agent_performance, ruvector_adaptive_route
- ruvector_fastgrnn_forward

### Learning/ReasoningBank (7 functions)
- ruvector_record_trajectory, ruvector_get_verdict
- ruvector_distill_memory, ruvector_adaptive_search
- ruvector_learning_feedback, ruvector_get_learning_patterns
- ruvector_optimize_search_params

### Graph/Cypher (8 functions)
- ruvector_graph_create_node, ruvector_graph_create_edge
- ruvector_graph_get_neighbors, ruvector_graph_shortest_path
- ruvector_graph_pagerank, ruvector_cypher_query
- ruvector_graph_traverse, ruvector_graph_similarity_search

## CLI Updates
- Enabled hyperbolic geometry commands in postgres-cli
- Added vector distance and normalize commands
- Enhanced client with connection pooling and retry logic

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-12-02 22:49:29 -05:00

6.7 KiB

RuVector-Postgres Docker Infrastructure

Docker-based development and testing environment for the ruvector-postgres PostgreSQL extension.

Quick Start

Development Environment

# Start development environment
./dev.sh start

# Open psql shell
./dev.sh psql

# Watch for changes and auto-reload
./dev.sh watch

# Stop environment
./dev.sh stop

Running Tests

# Run full test suite
./run-tests.sh

# Run integration tests only
./run-tests.sh --integration

# Keep container running for debugging
./run-tests.sh --keep-running

# Clean rebuild
./run-tests.sh --clean

Scripts Overview

dev.sh - Development Environment

Manages a PostgreSQL development environment with hot-reload support.

Commands:

  • start - Start development environment (default)
  • stop - Stop development environment
  • restart - Restart development environment
  • logs - Show PostgreSQL logs
  • psql - Open psql shell
  • watch - Start file watcher for hot-reload (requires cargo-watch)
  • rebuild - Rebuild and reload extension
  • status - Show container status

Options:

  • -p, --port PORT - PostgreSQL port (default: 5432)
  • -u, --user USER - PostgreSQL user (default: postgres)
  • -d, --database DB - PostgreSQL database (default: ruvector_dev)
  • -f, --foreground - Start in foreground with logs
  • -h, --help - Show help message

Examples:

# Start on custom port
./dev.sh --port 5433 start

# View logs
./dev.sh logs

# Rebuild extension
./dev.sh rebuild

run-tests.sh - Test Runner

Builds Docker image, runs tests, and manages test infrastructure.

Options:

  • -b, --build-only - Build Docker image only, don't run tests
  • -t, --test-only - Run tests only (skip build)
  • -i, --integration - Run integration tests only
  • -k, --keep-running - Keep container running after tests
  • -c, --clean - Clean up before starting
  • -v, --keep-volumes - Keep volumes after cleanup
  • -p, --port PORT - PostgreSQL port (default: 5433)
  • -h, --help - Show help message

Examples:

# Build and test
./run-tests.sh

# Integration tests with container kept running
./run-tests.sh --integration --keep-running

# Clean rebuild
./run-tests.sh --clean --build-only

Docker Files

Dockerfile - Main Build File

Multi-stage Docker build for PostgreSQL 16 with pgrx 0.12.6 support.

Features:

  • Rust 1.75 with Bookworm base
  • PostgreSQL 16 with development headers
  • cargo-pgrx 0.12.6 pre-installed
  • Optimized layer caching for dependencies
  • Health checks built-in

docker-compose.yml - Orchestration

Complete development stack with PostgreSQL and pgAdmin.

Services:

  • postgres - PostgreSQL 16 with ruvector extension
  • pgadmin - Web-based database management (port 5050)

Usage:

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

# Access pgAdmin
# URL: http://localhost:5050
# Email: admin@ruvector.dev
# Password: admin

init.sql - Database Initialization

SQL script for automatic database setup with:

  • Extension creation
  • Sample tables and indexes
  • Test data
  • Performance monitoring views

Development Workflow

1. Initial Setup

# Start development environment
./dev.sh start

# This will:
# - Pull PostgreSQL 16 image
# - Create development database
# - Expose on localhost:5432
# - Show connection string

2. Build Extension

cd /workspaces/ruvector/crates/ruvector-postgres

# Build and install extension
cargo pgrx install --release

3. Test Changes

# Quick test in psql
./dev.sh psql

# In psql:
# CREATE EXTENSION ruvector_postgres;
# SELECT '[1,2,3]'::vector;

4. Hot-Reload Development

# Install cargo-watch (one time)
cargo install cargo-watch

# Start watching for changes
./dev.sh watch

# Now edit code - extension auto-reloads on save!

5. Run Full Test Suite

# Run all tests
./run-tests.sh

# Or run just integration tests
./run-tests.sh --integration

Environment Variables

Development (dev.sh)

POSTGRES_PORT=5432          # PostgreSQL port
POSTGRES_USER=postgres      # PostgreSQL user
POSTGRES_PASSWORD=postgres  # PostgreSQL password
POSTGRES_DB=ruvector_dev    # Database name

Testing (run-tests.sh)

POSTGRES_PORT=5433          # PostgreSQL port (different from dev)
POSTGRES_USER=ruvector      # PostgreSQL user
POSTGRES_PASSWORD=ruvector  # PostgreSQL password
POSTGRES_DB=ruvector_test   # Test database name
KEEP_VOLUMES=false          # Keep volumes after cleanup
EXPORT_DB=false             # Export database dump

Platform Support

Both scripts support:

  • Linux (Ubuntu, Debian, RHEL, etc.)
  • macOS (Intel and Apple Silicon)
  • Windows (via WSL2)

The scripts automatically detect the platform and adjust behavior accordingly.

Troubleshooting

Port Already in Use

# Check what's using the port
lsof -i :5432

# Use a different port
./dev.sh --port 5433 start

Extension Not Loading

# Rebuild extension
./dev.sh rebuild

# Or manually:
cd /workspaces/ruvector/crates/ruvector-postgres
cargo pgrx install --release

# Then reload in database
./dev.sh psql
# DROP EXTENSION ruvector_postgres CASCADE;
# CREATE EXTENSION ruvector_postgres;

Docker Build Fails

# Clean build
docker system prune -a
./run-tests.sh --clean --build-only

# Check Docker resources
docker info

Tests Fail

# Keep container running to debug
./run-tests.sh --keep-running

# Connect to inspect
./dev.sh psql

# View logs
docker logs ruvector-postgres-test

Performance Tips

Build Optimization

# Use BuildKit for faster builds
export DOCKER_BUILDKIT=1
./run-tests.sh

# Parallel builds
docker build --build-arg MAKEFLAGS="-j$(nproc)" ...

Development Speed

# Use cargo-watch for instant feedback
./dev.sh watch

# Or use cargo-pgrx run for interactive development
cd /workspaces/ruvector/crates/ruvector-postgres
cargo pgrx run pg16

CI/CD Integration

GitHub Actions Example

name: Test RuVector-Postgres

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run tests
        run: |
          cd crates/ruvector-postgres/docker
          ./run-tests.sh

GitLab CI Example

test:
  image: docker:latest
  services:
    - docker:dind
  script:
    - cd crates/ruvector-postgres/docker
    - ./run-tests.sh

Resources

License

MIT License - See project root for details