feat(ai-memory-engine): Add 8 vector storage backends + hyperbolic geometry

Storage Backends:
- ApifyBinary: 4x smaller binary Float32Array storage
- PostgreSQL/Supabase/Neon: pgvector with IVFFlat indexing
- Qdrant: High-throughput cloud vector DB with filtering
- Pinecone: Serverless managed infrastructure
- Weaviate: GraphQL-based hybrid search
- LanceDB: Local embedded vector database
- Hyperbolic: Poincaré ball model for hierarchical data

Hyperbolic Geometry:
- Poincaré ball projection and distance
- Möbius addition for vector operations
- Exponential/logarithmic maps (tangent ↔ hyperbolic)
- Fréchet mean (hyperbolic centroid)
- Hyperbolic k-means clustering
- Batch distance computation (SIMD-friendly)
- Parallel transport and geodesic interpolation

Input Schema:
- storageBackend selector (8 options)
- Connection strings for Postgres/Qdrant/Pinecone/Weaviate
- Curvature parameter for hyperbolic embeddings
- Hyperbolic clustering toggle

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
rUv 2025-12-14 04:12:06 +00:00
parent c9581a7fa6
commit fbf985dc40
3 changed files with 1267 additions and 2 deletions

View file

@ -762,6 +762,103 @@
"description": "Set a seed for reproducible results across runs",
"editor": "textfield",
"example": "42"
},
"storageBackend": {
"title": "Storage Backend",
"type": "string",
"description": "Choose how and where to persist your memory embeddings. Different backends offer different trade-offs for scalability, persistence, and features.",
"editor": "select",
"default": "apify-binary",
"sectionCaption": "Vector Storage & Persistence",
"sectionDescription": "Configure advanced vector storage options. Choose from 8 backends including cloud vector databases, hyperbolic embeddings, and local storage.",
"enum": [
"apify-binary",
"postgres",
"supabase",
"neon",
"qdrant",
"pinecone",
"weaviate",
"lancedb",
"hyperbolic"
],
"enumTitles": [
"Apify Binary - Default, 4x smaller than JSON (recommended)",
"PostgreSQL - Self-hosted with pgvector extension",
"Supabase - Managed PostgreSQL with pgvector",
"Neon - Serverless PostgreSQL with pgvector",
"Qdrant - High-performance vector database",
"Pinecone - Serverless managed vector DB",
"Weaviate - GraphQL-based vector search",
"LanceDB - Local embedded vector DB",
"Hyperbolic - Poincaré ball for hierarchical data"
]
},
"postgresUrl": {
"title": "PostgreSQL Connection URL",
"type": "string",
"description": "Connection string for PostgreSQL/Supabase/Neon (e.g., postgres://user:pass@host:5432/db)",
"editor": "textfield",
"isSecret": true,
"example": "postgres://user:password@db.supabase.co:5432/postgres"
},
"qdrantUrl": {
"title": "Qdrant URL",
"type": "string",
"description": "Qdrant server URL (cloud or self-hosted)",
"editor": "textfield",
"default": "http://localhost:6333",
"example": "https://your-cluster.qdrant.io"
},
"qdrantApiKey": {
"title": "Qdrant API Key",
"type": "string",
"description": "API key for Qdrant Cloud authentication",
"editor": "textfield",
"isSecret": true
},
"pineconeApiKey": {
"title": "Pinecone API Key",
"type": "string",
"description": "API key for Pinecone serverless",
"editor": "textfield",
"isSecret": true
},
"pineconeIndex": {
"title": "Pinecone Index Name",
"type": "string",
"description": "Name of your Pinecone index",
"editor": "textfield",
"default": "ai-memories"
},
"weaviateHost": {
"title": "Weaviate Host",
"type": "string",
"description": "Weaviate instance host (without https://)",
"editor": "textfield",
"example": "your-instance.weaviate.network"
},
"weaviateApiKey": {
"title": "Weaviate API Key",
"type": "string",
"description": "API key for Weaviate Cloud authentication",
"editor": "textfield",
"isSecret": true
},
"curvature": {
"title": "Hyperbolic Curvature",
"type": "number",
"description": "Curvature parameter for Poincaré ball model (higher = more curved, better for deep hierarchies)",
"default": 1.0,
"minimum": 0.1,
"maximum": 10.0
},
"hyperbolicClustering": {
"title": "Enable Hyperbolic Clustering",
"type": "boolean",
"description": "Use hyperbolic k-means for hierarchical data clustering (better for tree-like relationships)",
"default": false
}
},
"required": ["action"]

View file

@ -2,7 +2,7 @@
**Give your AI persistent memory.** Store conversations, search semantically, build knowledge graphs, and watch your AI get smarter with every interaction. The ultimate memory solution for AI agents, chatbots, and intelligent automation.
🧠 **Self-Learning** · 🔍 **Semantic Search** · 🕸️ **Knowledge Graphs** · ⚡ **Sub-millisecond** · 🔗 **LLM Agnostic** · 🔌 **One-Click Integrations**
🧠 **Self-Learning** · 🔍 **Semantic Search** · 🕸️ **Knowledge Graphs** · ⚡ **Sub-millisecond** · 🔗 **LLM Agnostic** · 🔌 **One-Click Integrations** · 📊 **8 Vector DB Backends** · 🌀 **Hyperbolic Embeddings**
[![Apify Actor](https://img.shields.io/badge/Apify-Actor-blue)](https://apify.com/ruv/ai-memory-engine)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
@ -21,9 +21,10 @@
|---------|------------------|----------------------|
| **Setup Time** | 1 minute | Hours to days |
| **Cost** | Pay per use ($0.001/operation) | $50-500/month subscriptions |
| **Memory Persistence** | Built-in cross-session | Requires external DB |
| **Memory Persistence** | 8 storage backends (Supabase, Qdrant, Pinecone, etc.) | Single DB only |
| **Self-Learning** | SONA neural architecture | Manual tuning |
| **Apify Integration** | One-click from any actor | Custom code required |
| **Hyperbolic Geometry** | Poincaré ball for hierarchical data | Euclidean only |
| **Export Options** | 6 vector DB formats | Limited |
### Key Capabilities
@ -35,6 +36,15 @@
- **Integration**: integrate_actor, integrate_synthetic, integrate_scraper, template
- **Utilities**: natural language commands, cluster, export_vectordb, feedback
**8 Storage Backends**:
- **Apify Binary** - Default, 4x smaller than JSON with HNSW re-indexing
- **PostgreSQL/Supabase/Neon** - pgvector for production deployments
- **Qdrant** - High-throughput cloud vector DB
- **Pinecone** - Serverless managed infrastructure
- **Weaviate** - GraphQL-based hybrid search
- **LanceDB** - Local embedded database
- **Hyperbolic** - Poincaré ball for hierarchical data
**One-Click Actor Integration** with 10+ popular scrapers:
- Google Maps, Instagram, TikTok, YouTube, Twitter, Amazon, TripAdvisor, LinkedIn, and more
- Automatically memorize any scraper results with semantic search
@ -503,6 +513,134 @@ Each template includes:
---
## Vector Storage Backends
**8 storage backends** for different scalability and persistence needs:
```json
{
"action": "store",
"memories": [...],
"storageBackend": "supabase",
"postgresUrl": "postgres://user:pass@db.supabase.co:5432/postgres"
}
```
### Available Backends
| Backend | Best For | Features |
|---------|----------|----------|
| `apify-binary` | Default, simple persistence | 4x smaller than JSON, automatic HNSW re-indexing |
| `postgres` / `supabase` / `neon` | Production deployments | pgvector extension, SQL queries, ACID compliance |
| `qdrant` | High-throughput search | Built-in filtering, clustering, 100M+ vectors |
| `pinecone` | Serverless, zero ops | Managed infrastructure, auto-scaling |
| `weaviate` | GraphQL queries | Hybrid search, graph traversal |
| `lancedb` | Local/embedded | No server needed, ML-pipeline friendly |
| `hyperbolic` | Hierarchical data | Poincaré ball embeddings, tree structures |
### PostgreSQL / Supabase / Neon
Use pgvector for production-grade vector search:
```json
{
"action": "store",
"memories": [{"text": "Product X specs..."}],
"storageBackend": "supabase",
"postgresUrl": "postgres://user:pass@db.supabase.co:5432/postgres",
"sessionId": "my-app"
}
```
Features:
- IVFFlat and HNSW indexing
- SQL filtering with vector search
- Automatic schema creation
- Incremental updates (upsert)
### Qdrant Cloud
High-performance vector search at scale:
```json
{
"action": "store",
"storageBackend": "qdrant",
"qdrantUrl": "https://your-cluster.qdrant.io",
"qdrantApiKey": "your-api-key"
}
```
### Pinecone Serverless
Zero-ops managed vector database:
```json
{
"action": "store",
"storageBackend": "pinecone",
"pineconeApiKey": "your-api-key",
"pineconeIndex": "my-memories"
}
```
---
## Hyperbolic Embeddings
**NEW: Poincaré ball embeddings** for hierarchical and tree-structured data:
```json
{
"action": "store",
"memories": [
{"text": "CEO leads the company", "metadata": {"level": 1}},
{"text": "VP Engineering reports to CEO", "metadata": {"level": 2}},
{"text": "Senior Engineer reports to VP", "metadata": {"level": 3}}
],
"storageBackend": "hyperbolic",
"curvature": 1.0
}
```
### Why Hyperbolic?
Traditional Euclidean embeddings struggle with:
- **Taxonomies** (products → categories → subcategories)
- **Organization charts** (CEO → VPs → Managers → Staff)
- **Knowledge graphs** (concepts → related concepts)
- **File systems** (directories → subdirectories → files)
Hyperbolic space naturally represents hierarchical relationships because:
- Distance from origin = depth in hierarchy
- Points near boundary = leaf nodes
- Points near center = root nodes
### Hyperbolic Features
| Feature | Description |
|---------|-------------|
| `poincareDistance` | True geodesic distance in curved space |
| `mobiusAdd` | Addition operation preserving geometry |
| `frechetMean` | Hyperbolic centroid computation |
| `hyperbolicKMeans` | Clustering respecting hierarchy |
| `expMap` / `logMap` | Tangent space operations |
### Hyperbolic Clustering
Group hierarchical data naturally:
```json
{
"action": "cluster",
"numClusters": 5,
"hyperbolicClustering": true,
"curvature": 1.0
}
```
---
## Vector DB Export
**Export to any vector database:**

File diff suppressed because it is too large Load diff