mirror of
https://github.com/ruvnet/RuVector.git
synced 2026-05-24 22:15:18 +00:00
feat(ai-memory-engine): Add 8 vector storage backends + hyperbolic geometry
Storage Backends: - ApifyBinary: 4x smaller binary Float32Array storage - PostgreSQL/Supabase/Neon: pgvector with IVFFlat indexing - Qdrant: High-throughput cloud vector DB with filtering - Pinecone: Serverless managed infrastructure - Weaviate: GraphQL-based hybrid search - LanceDB: Local embedded vector database - Hyperbolic: Poincaré ball model for hierarchical data Hyperbolic Geometry: - Poincaré ball projection and distance - Möbius addition for vector operations - Exponential/logarithmic maps (tangent ↔ hyperbolic) - Fréchet mean (hyperbolic centroid) - Hyperbolic k-means clustering - Batch distance computation (SIMD-friendly) - Parallel transport and geodesic interpolation Input Schema: - storageBackend selector (8 options) - Connection strings for Postgres/Qdrant/Pinecone/Weaviate - Curvature parameter for hyperbolic embeddings - Hyperbolic clustering toggle 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
c9581a7fa6
commit
fbf985dc40
3 changed files with 1267 additions and 2 deletions
|
|
@ -762,6 +762,103 @@
|
|||
"description": "Set a seed for reproducible results across runs",
|
||||
"editor": "textfield",
|
||||
"example": "42"
|
||||
},
|
||||
|
||||
"storageBackend": {
|
||||
"title": "Storage Backend",
|
||||
"type": "string",
|
||||
"description": "Choose how and where to persist your memory embeddings. Different backends offer different trade-offs for scalability, persistence, and features.",
|
||||
"editor": "select",
|
||||
"default": "apify-binary",
|
||||
"sectionCaption": "Vector Storage & Persistence",
|
||||
"sectionDescription": "Configure advanced vector storage options. Choose from 8 backends including cloud vector databases, hyperbolic embeddings, and local storage.",
|
||||
"enum": [
|
||||
"apify-binary",
|
||||
"postgres",
|
||||
"supabase",
|
||||
"neon",
|
||||
"qdrant",
|
||||
"pinecone",
|
||||
"weaviate",
|
||||
"lancedb",
|
||||
"hyperbolic"
|
||||
],
|
||||
"enumTitles": [
|
||||
"Apify Binary - Default, 4x smaller than JSON (recommended)",
|
||||
"PostgreSQL - Self-hosted with pgvector extension",
|
||||
"Supabase - Managed PostgreSQL with pgvector",
|
||||
"Neon - Serverless PostgreSQL with pgvector",
|
||||
"Qdrant - High-performance vector database",
|
||||
"Pinecone - Serverless managed vector DB",
|
||||
"Weaviate - GraphQL-based vector search",
|
||||
"LanceDB - Local embedded vector DB",
|
||||
"Hyperbolic - Poincaré ball for hierarchical data"
|
||||
]
|
||||
},
|
||||
"postgresUrl": {
|
||||
"title": "PostgreSQL Connection URL",
|
||||
"type": "string",
|
||||
"description": "Connection string for PostgreSQL/Supabase/Neon (e.g., postgres://user:pass@host:5432/db)",
|
||||
"editor": "textfield",
|
||||
"isSecret": true,
|
||||
"example": "postgres://user:password@db.supabase.co:5432/postgres"
|
||||
},
|
||||
"qdrantUrl": {
|
||||
"title": "Qdrant URL",
|
||||
"type": "string",
|
||||
"description": "Qdrant server URL (cloud or self-hosted)",
|
||||
"editor": "textfield",
|
||||
"default": "http://localhost:6333",
|
||||
"example": "https://your-cluster.qdrant.io"
|
||||
},
|
||||
"qdrantApiKey": {
|
||||
"title": "Qdrant API Key",
|
||||
"type": "string",
|
||||
"description": "API key for Qdrant Cloud authentication",
|
||||
"editor": "textfield",
|
||||
"isSecret": true
|
||||
},
|
||||
"pineconeApiKey": {
|
||||
"title": "Pinecone API Key",
|
||||
"type": "string",
|
||||
"description": "API key for Pinecone serverless",
|
||||
"editor": "textfield",
|
||||
"isSecret": true
|
||||
},
|
||||
"pineconeIndex": {
|
||||
"title": "Pinecone Index Name",
|
||||
"type": "string",
|
||||
"description": "Name of your Pinecone index",
|
||||
"editor": "textfield",
|
||||
"default": "ai-memories"
|
||||
},
|
||||
"weaviateHost": {
|
||||
"title": "Weaviate Host",
|
||||
"type": "string",
|
||||
"description": "Weaviate instance host (without https://)",
|
||||
"editor": "textfield",
|
||||
"example": "your-instance.weaviate.network"
|
||||
},
|
||||
"weaviateApiKey": {
|
||||
"title": "Weaviate API Key",
|
||||
"type": "string",
|
||||
"description": "API key for Weaviate Cloud authentication",
|
||||
"editor": "textfield",
|
||||
"isSecret": true
|
||||
},
|
||||
"curvature": {
|
||||
"title": "Hyperbolic Curvature",
|
||||
"type": "number",
|
||||
"description": "Curvature parameter for Poincaré ball model (higher = more curved, better for deep hierarchies)",
|
||||
"default": 1.0,
|
||||
"minimum": 0.1,
|
||||
"maximum": 10.0
|
||||
},
|
||||
"hyperbolicClustering": {
|
||||
"title": "Enable Hyperbolic Clustering",
|
||||
"type": "boolean",
|
||||
"description": "Use hyperbolic k-means for hierarchical data clustering (better for tree-like relationships)",
|
||||
"default": false
|
||||
}
|
||||
},
|
||||
"required": ["action"]
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
**Give your AI persistent memory.** Store conversations, search semantically, build knowledge graphs, and watch your AI get smarter with every interaction. The ultimate memory solution for AI agents, chatbots, and intelligent automation.
|
||||
|
||||
🧠 **Self-Learning** · 🔍 **Semantic Search** · 🕸️ **Knowledge Graphs** · ⚡ **Sub-millisecond** · 🔗 **LLM Agnostic** · 🔌 **One-Click Integrations**
|
||||
🧠 **Self-Learning** · 🔍 **Semantic Search** · 🕸️ **Knowledge Graphs** · ⚡ **Sub-millisecond** · 🔗 **LLM Agnostic** · 🔌 **One-Click Integrations** · 📊 **8 Vector DB Backends** · 🌀 **Hyperbolic Embeddings**
|
||||
|
||||
[](https://apify.com/ruv/ai-memory-engine)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
|
|
@ -21,9 +21,10 @@
|
|||
|---------|------------------|----------------------|
|
||||
| **Setup Time** | 1 minute | Hours to days |
|
||||
| **Cost** | Pay per use ($0.001/operation) | $50-500/month subscriptions |
|
||||
| **Memory Persistence** | Built-in cross-session | Requires external DB |
|
||||
| **Memory Persistence** | 8 storage backends (Supabase, Qdrant, Pinecone, etc.) | Single DB only |
|
||||
| **Self-Learning** | SONA neural architecture | Manual tuning |
|
||||
| **Apify Integration** | One-click from any actor | Custom code required |
|
||||
| **Hyperbolic Geometry** | Poincaré ball for hierarchical data | Euclidean only |
|
||||
| **Export Options** | 6 vector DB formats | Limited |
|
||||
|
||||
### Key Capabilities
|
||||
|
|
@ -35,6 +36,15 @@
|
|||
- **Integration**: integrate_actor, integrate_synthetic, integrate_scraper, template
|
||||
- **Utilities**: natural language commands, cluster, export_vectordb, feedback
|
||||
|
||||
**8 Storage Backends**:
|
||||
- **Apify Binary** - Default, 4x smaller than JSON with HNSW re-indexing
|
||||
- **PostgreSQL/Supabase/Neon** - pgvector for production deployments
|
||||
- **Qdrant** - High-throughput cloud vector DB
|
||||
- **Pinecone** - Serverless managed infrastructure
|
||||
- **Weaviate** - GraphQL-based hybrid search
|
||||
- **LanceDB** - Local embedded database
|
||||
- **Hyperbolic** - Poincaré ball for hierarchical data
|
||||
|
||||
**One-Click Actor Integration** with 10+ popular scrapers:
|
||||
- Google Maps, Instagram, TikTok, YouTube, Twitter, Amazon, TripAdvisor, LinkedIn, and more
|
||||
- Automatically memorize any scraper results with semantic search
|
||||
|
|
@ -503,6 +513,134 @@ Each template includes:
|
|||
|
||||
---
|
||||
|
||||
## Vector Storage Backends
|
||||
|
||||
**8 storage backends** for different scalability and persistence needs:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "store",
|
||||
"memories": [...],
|
||||
"storageBackend": "supabase",
|
||||
"postgresUrl": "postgres://user:pass@db.supabase.co:5432/postgres"
|
||||
}
|
||||
```
|
||||
|
||||
### Available Backends
|
||||
|
||||
| Backend | Best For | Features |
|
||||
|---------|----------|----------|
|
||||
| `apify-binary` | Default, simple persistence | 4x smaller than JSON, automatic HNSW re-indexing |
|
||||
| `postgres` / `supabase` / `neon` | Production deployments | pgvector extension, SQL queries, ACID compliance |
|
||||
| `qdrant` | High-throughput search | Built-in filtering, clustering, 100M+ vectors |
|
||||
| `pinecone` | Serverless, zero ops | Managed infrastructure, auto-scaling |
|
||||
| `weaviate` | GraphQL queries | Hybrid search, graph traversal |
|
||||
| `lancedb` | Local/embedded | No server needed, ML-pipeline friendly |
|
||||
| `hyperbolic` | Hierarchical data | Poincaré ball embeddings, tree structures |
|
||||
|
||||
### PostgreSQL / Supabase / Neon
|
||||
|
||||
Use pgvector for production-grade vector search:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "store",
|
||||
"memories": [{"text": "Product X specs..."}],
|
||||
"storageBackend": "supabase",
|
||||
"postgresUrl": "postgres://user:pass@db.supabase.co:5432/postgres",
|
||||
"sessionId": "my-app"
|
||||
}
|
||||
```
|
||||
|
||||
Features:
|
||||
- IVFFlat and HNSW indexing
|
||||
- SQL filtering with vector search
|
||||
- Automatic schema creation
|
||||
- Incremental updates (upsert)
|
||||
|
||||
### Qdrant Cloud
|
||||
|
||||
High-performance vector search at scale:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "store",
|
||||
"storageBackend": "qdrant",
|
||||
"qdrantUrl": "https://your-cluster.qdrant.io",
|
||||
"qdrantApiKey": "your-api-key"
|
||||
}
|
||||
```
|
||||
|
||||
### Pinecone Serverless
|
||||
|
||||
Zero-ops managed vector database:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "store",
|
||||
"storageBackend": "pinecone",
|
||||
"pineconeApiKey": "your-api-key",
|
||||
"pineconeIndex": "my-memories"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Hyperbolic Embeddings
|
||||
|
||||
**NEW: Poincaré ball embeddings** for hierarchical and tree-structured data:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "store",
|
||||
"memories": [
|
||||
{"text": "CEO leads the company", "metadata": {"level": 1}},
|
||||
{"text": "VP Engineering reports to CEO", "metadata": {"level": 2}},
|
||||
{"text": "Senior Engineer reports to VP", "metadata": {"level": 3}}
|
||||
],
|
||||
"storageBackend": "hyperbolic",
|
||||
"curvature": 1.0
|
||||
}
|
||||
```
|
||||
|
||||
### Why Hyperbolic?
|
||||
|
||||
Traditional Euclidean embeddings struggle with:
|
||||
- **Taxonomies** (products → categories → subcategories)
|
||||
- **Organization charts** (CEO → VPs → Managers → Staff)
|
||||
- **Knowledge graphs** (concepts → related concepts)
|
||||
- **File systems** (directories → subdirectories → files)
|
||||
|
||||
Hyperbolic space naturally represents hierarchical relationships because:
|
||||
- Distance from origin = depth in hierarchy
|
||||
- Points near boundary = leaf nodes
|
||||
- Points near center = root nodes
|
||||
|
||||
### Hyperbolic Features
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `poincareDistance` | True geodesic distance in curved space |
|
||||
| `mobiusAdd` | Addition operation preserving geometry |
|
||||
| `frechetMean` | Hyperbolic centroid computation |
|
||||
| `hyperbolicKMeans` | Clustering respecting hierarchy |
|
||||
| `expMap` / `logMap` | Tangent space operations |
|
||||
|
||||
### Hyperbolic Clustering
|
||||
|
||||
Group hierarchical data naturally:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "cluster",
|
||||
"numClusters": 5,
|
||||
"hyperbolicClustering": true,
|
||||
"curvature": 1.0
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Vector DB Export
|
||||
|
||||
**Export to any vector database:**
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue