mirror of
https://github.com/supermemoryai/supermemory.git
synced 2026-08-25 16:32:45 +00:00
Adds a focused guide for backfilling dated documents with `documentDate` and the batch ingestion API. - includes TypeScript and Python batch examples plus optional completion polling - links the guide from the docs navigation and ingestion entry points Validated with `bunx mintlify@latest validate` and `bunx mintlify@latest broken-links`.
502 lines
15 KiB
Text
502 lines
15 KiB
Text
---
|
|
title: "Ingesting context to supermemory"
|
|
sidebarTitle: "API"
|
|
description: "Add text, files, and URLs to Supermemory"
|
|
icon: "plus"
|
|
---
|
|
|
|
Send any raw content to Supermemory — conversations, documents, files, URLs. We extract the memories automatically. Pass `customId` to identify content and avoid duplicates, and `taskType: "superrag"` if you just need it searchable, not remembered — that's [5x cheaper](#memory-vs-superrag-ingestion) per token.
|
|
|
|
## Quick Start
|
|
|
|
<Tabs>
|
|
<Tab title="TypeScript">
|
|
```typescript
|
|
import Supermemory from 'supermemory';
|
|
|
|
const client = new Supermemory();
|
|
|
|
// Add text content
|
|
await client.add({
|
|
content: "Machine learning enables computers to learn from data",
|
|
containerTag: "user_123",
|
|
metadata: { category: "ai" }
|
|
});
|
|
|
|
// Add a URL (auto-extracted)
|
|
await client.add({
|
|
content: "https://youtube.com/watch?v=dQw4w9WgXcQ",
|
|
containerTag: "user_123"
|
|
});
|
|
```
|
|
</Tab>
|
|
<Tab title="Python">
|
|
```python
|
|
from supermemory import Supermemory
|
|
|
|
client = Supermemory()
|
|
|
|
# Add text content
|
|
client.add(
|
|
content="Machine learning enables computers to learn from data",
|
|
container_tag="user_123",
|
|
metadata={"category": "ai"}
|
|
)
|
|
|
|
# Add a URL (auto-extracted)
|
|
client.add(
|
|
content="https://youtube.com/watch?v=dQw4w9WgXcQ",
|
|
container_tag="user_123"
|
|
)
|
|
```
|
|
</Tab>
|
|
<Tab title="cURL">
|
|
```bash
|
|
curl -X POST "https://api.supermemory.ai/v3/documents" \
|
|
-H "Authorization: Bearer $SUPERMEMORY_API_KEY" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"content": "Machine learning enables computers to learn from data",
|
|
"containerTag": "user_123",
|
|
"metadata": {"category": "ai"}
|
|
}'
|
|
```
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
**Response:**
|
|
```json
|
|
{ "id": "abc123", "status": "queued" }
|
|
```
|
|
|
|
<Warning>
|
|
If an irrecoverable processing error occurs, the document is automatically deleted after 2 minutes.
|
|
</Warning>
|
|
|
|
---
|
|
|
|
## Updating Content
|
|
|
|
Use `customId` to update existing documents or conversations. When you send content with the same `customId`, Supermemory intelligently processes only what's new.
|
|
|
|
### Two ways to update:
|
|
|
|
**Option 1: Send only the new content**
|
|
```typescript
|
|
// First request
|
|
await client.add({
|
|
content: "user: Hi, I'm Sarah.\nassistant: Nice to meet you!",
|
|
customId: "conv_123",
|
|
containerTag: "user_sarah"
|
|
});
|
|
|
|
// Later: send only new messages
|
|
await client.add({
|
|
content: "user: What's the weather?\nassistant: It's sunny today.",
|
|
customId: "conv_123", // Same ID — Supermemory links them
|
|
containerTag: "user_sarah"
|
|
});
|
|
```
|
|
|
|
**Option 2: Send the full updated content**
|
|
```typescript
|
|
// Supermemory detects the diff and only processes new parts
|
|
await client.add({
|
|
content: "user: Hi, I'm Sarah.\nassistant: Nice to meet you!\nuser: What's the weather?\nassistant: It's sunny today.",
|
|
customId: "conv_123",
|
|
containerTag: "user_sarah"
|
|
});
|
|
```
|
|
|
|
Both work — choose what fits your architecture.
|
|
|
|
### Replace entire document
|
|
|
|
To completely replace a document's content (not append), use `memories.update()`:
|
|
|
|
```typescript
|
|
// Replace the entire document content
|
|
await client.documents.update("doc_id_123", {
|
|
content: "Completely new content replacing everything",
|
|
metadata: { version: 2 }
|
|
});
|
|
```
|
|
|
|
This triggers full reprocessing of the document. If you only update metadata (no content change), the document is updated in place with no reindexing.
|
|
|
|
### Formatting conversations
|
|
|
|
Format your conversations however you want. Supermemory handles any string format:
|
|
|
|
```typescript
|
|
// Simple string
|
|
content: "user: Hello\nassistant: Hi there!"
|
|
|
|
// JSON stringify
|
|
content: JSON.stringify(messages)
|
|
|
|
// Template literal
|
|
content: messages.map(m => `${m.role}: ${m.content}`).join('\n')
|
|
|
|
// Any format — just make it a string
|
|
content: formatConversation(messages)
|
|
```
|
|
|
|
---
|
|
|
|
## Upload Files
|
|
|
|
Upload PDFs, images, and documents directly.
|
|
|
|
<Tabs>
|
|
<Tab title="TypeScript">
|
|
```typescript
|
|
import fs from 'fs';
|
|
|
|
await client.documents.uploadFile({
|
|
file: fs.createReadStream('document.pdf'),
|
|
containerTag: 'user_123'
|
|
});
|
|
```
|
|
</Tab>
|
|
<Tab title="Python">
|
|
```python
|
|
with open('document.pdf', 'rb') as file:
|
|
client.documents.upload_file(
|
|
file=file,
|
|
container_tag='user_123'
|
|
)
|
|
```
|
|
</Tab>
|
|
<Tab title="cURL">
|
|
```bash
|
|
curl -X POST "https://api.supermemory.ai/v3/documents/file" \
|
|
-H "Authorization: Bearer $SUPERMEMORY_API_KEY" \
|
|
-F "file=@document.pdf" \
|
|
-F "containerTag=user_123"
|
|
```
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
### Supported File Types
|
|
|
|
| Type | Formats | Processing |
|
|
|------|---------|------------|
|
|
| Documents | PDF, DOC, DOCX, TXT, MD | Text extraction, OCR for scans |
|
|
| Images | JPG, PNG, GIF, WebP | OCR text extraction |
|
|
| Spreadsheets | CSV, Google Sheets | Structured data extraction |
|
|
| Videos | YouTube URLs, MP4 | Auto-transcription |
|
|
|
|
**Limits:** 50MB max file size
|
|
|
|
---
|
|
|
|
## Parameters
|
|
|
|
| Parameter | Type | Description |
|
|
|-----------|------|-------------|
|
|
| `content` | string | **Required.** Any raw content — text, conversations, URLs, HTML |
|
|
| `customId` | string | **Recommended.** Your ID for the content (conversation ID, doc ID). Enables updates and deduplication |
|
|
| `containerTag` | string | Group by user/project. Required for user profiles |
|
|
| `metadata` | object | Key-value pairs for filtering (strings, numbers, booleans) |
|
|
| `filterByMetadata` | object | Filter which existing memories are used as context during ingestion. See [Filtered Writes](#filtered-writes) |
|
|
| `entityContext` | string | Context for memory extraction on this container tag. Max 1500 chars. See [Customization](/concepts/customization#entity-context) |
|
|
| `dreaming` | `"dynamic" \| "instant"` | Processing mode. Default `"dynamic"`. `"instant"` processes each document on its own and bills one extra operation. See [Processing Modes](#processing-modes) |
|
|
| `taskType` | `"memory" \| "superrag"` | Pipeline to run. Default `"memory"`. `"superrag"` skips fact extraction and profile updates, doing only chunk/embed/index — at 5x cheaper per token. See [SuperRAG ingestion](/concepts/super-rag#ingesting-as-pure-superrag-tasktype-superrag) |
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="Parameter Details & Examples">
|
|
**Content Types:**
|
|
```typescript
|
|
// Any text — conversations, notes, documents
|
|
{ content: "Meeting notes from today's standup" }
|
|
{ content: JSON.stringify(messages) }
|
|
|
|
// URLs (auto-detected and extracted)
|
|
{ content: "https://example.com/article" }
|
|
{ content: "https://youtube.com/watch?v=abc123" }
|
|
|
|
// Markdown, HTML, or any format
|
|
{ content: "# Project Docs\n\n## Features\n- Real-time sync" }
|
|
```
|
|
|
|
**Container Tags:**
|
|
```typescript
|
|
// By user
|
|
{ containerTag: "user_123" }
|
|
|
|
// By project
|
|
{ containerTag: "project_alpha" }
|
|
|
|
// Hierarchical
|
|
{ containerTag: "org_456_team_backend" }
|
|
```
|
|
|
|
**Custom IDs (Recommended):**
|
|
```typescript
|
|
// Use IDs from your system
|
|
{ customId: "conv_abc123" } // Conversation ID
|
|
{ customId: "doc_456" } // Document ID
|
|
{ customId: "thread_789" } // Thread ID
|
|
{ customId: "meeting_2024_01_15" } // Meeting ID
|
|
|
|
// Updates: same customId = same document
|
|
// Supermemory only processes new/changed content
|
|
await client.add({
|
|
content: "Updated content...",
|
|
customId: "doc_456" // Links to existing document
|
|
});
|
|
```
|
|
|
|
**Metadata:**
|
|
```typescript
|
|
{
|
|
metadata: {
|
|
source: "slack",
|
|
author: "john",
|
|
priority: 1,
|
|
reviewed: true
|
|
}
|
|
}
|
|
```
|
|
- No nested objects or arrays
|
|
- Values: string, number, or boolean only
|
|
|
|
**Entity Context:**
|
|
```typescript
|
|
// Guide memory extraction for this container tag
|
|
{
|
|
containerTag: "session_abc123",
|
|
entityContext: `Design exploration conversation between john@acme.com and Brand.ai assistant.
|
|
Focus on John's design preferences and brand requirements.`
|
|
}
|
|
```
|
|
- Max 1500 characters
|
|
- Persists on the container tag
|
|
- Combines with org-level filter prompts
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
---
|
|
|
|
## Processing Modes
|
|
|
|
### Dreaming: dynamic vs instant
|
|
|
|
The `dreaming` parameter controls how Supermemory turns a document into memories.
|
|
|
|
- `"dynamic"` (default) — groups related documents together so memories form from coherent, logical units rather than one isolated entry at a time.
|
|
- `"instant"` — processes each document on its own right away, and bills one extra operation per document.
|
|
|
|
```json
|
|
{
|
|
"content": "...",
|
|
"dreaming": "instant"
|
|
}
|
|
```
|
|
|
|
### Memory vs SuperRAG ingestion
|
|
|
|
The `taskType` parameter controls whether that content also feeds the memory pipeline.
|
|
|
|
- `"memory"` (default) — chunks/embeds for search **and** extracts facts, updates the user's profile, and links into the graph.
|
|
- `"superrag"` — chunks/embeds for search only. No fact extraction, no profile updates. Priced at **5x cheaper per token** than `"memory"`.
|
|
|
|
```json
|
|
{
|
|
"content": "...",
|
|
"taskType": "superrag"
|
|
}
|
|
```
|
|
|
|
Use `"superrag"` for reference material you want searchable but that shouldn't shape what Supermemory knows about a user. Full explanation: [SuperRAG → Ingesting as pure SuperRAG](/concepts/super-rag#ingesting-as-pure-superrag-tasktype-superrag).
|
|
|
|
---
|
|
|
|
## Filtered Writes
|
|
|
|
By default, when you add content, Supermemory uses **all** existing memories in the space as context for generating new memories. With **filtered writes**, you can scope this context to only memories from documents matching specific metadata.
|
|
|
|
This is useful when you have many documents in a space but want new memories to build on top of a specific subset — for example, only memories from a particular source, category, or user.
|
|
|
|
<Note>
|
|
The metadata itself is still written to the document, but the memories will only be built on top of what's already there matching the filter.
|
|
</Note>
|
|
|
|
<Tabs>
|
|
<Tab title="TypeScript">
|
|
```typescript
|
|
await client.add({
|
|
content: "New research findings on transformer architectures...",
|
|
containerTag: "user_123",
|
|
metadata: { category: "ml", source: "arxiv" },
|
|
filterByMetadata: { category: "ml" }
|
|
});
|
|
```
|
|
</Tab>
|
|
<Tab title="Python">
|
|
```python
|
|
client.add(
|
|
content="New research findings on transformer architectures...",
|
|
container_tag="user_123",
|
|
metadata={"category": "ml", "source": "arxiv"},
|
|
filter_by_metadata={"category": "ml"}
|
|
)
|
|
```
|
|
</Tab>
|
|
<Tab title="cURL">
|
|
```bash
|
|
curl -X POST "https://api.supermemory.ai/v3/documents" \
|
|
-H "Authorization: Bearer $SUPERMEMORY_API_KEY" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"content": "New research findings on transformer architectures...",
|
|
"containerTag": "user_123",
|
|
"metadata": {"category": "ml", "source": "arxiv"},
|
|
"filterByMetadata": {"category": "ml"}
|
|
}'
|
|
```
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
### How it works
|
|
|
|
When `filterByMetadata` is provided:
|
|
- **Profile memories** (static context) are filtered to only those from documents matching the metadata
|
|
- **Similar memories** used as context during ingestion are filtered the same way
|
|
- The new document's own metadata is written normally — the filter only affects which **existing** memories are used as context
|
|
|
|
### `filterByMetadata` parameter
|
|
|
|
| Key | Type | Description |
|
|
|-----|------|-------------|
|
|
| `filterByMetadata` | `Record<string, string \| number \| boolean \| string[]>` | Key-value pairs to filter existing memories by their source document metadata |
|
|
|
|
- **Scalar values** (string, number, boolean) match exactly
|
|
- **Array values** match if **any** value in the array matches (OR logic)
|
|
- **Multiple keys** are combined with AND logic
|
|
|
|
```typescript
|
|
// Match documents where category is "ml" AND source is either "arxiv" or "pubmed"
|
|
await client.add({
|
|
content: "...",
|
|
containerTag: "user_123",
|
|
filterByMetadata: {
|
|
category: "ml",
|
|
source: ["arxiv", "pubmed"]
|
|
}
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## Processing Pipeline
|
|
|
|
When you add content, Supermemory:
|
|
|
|
1. **Validates** your request
|
|
2. **Stores** the document and queues for processing
|
|
3. **Extracts** content (OCR, transcription, web scraping)
|
|
4. **Chunks** into searchable memories
|
|
5. **Embeds** for vector search
|
|
6. **Indexes** for retrieval
|
|
|
|
Track progress with `GET /v3/documents/{id}`:
|
|
```typescript
|
|
const doc = await client.documents.get("abc123");
|
|
console.log(doc.status); // "queued" | "processing" | "done"
|
|
```
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="Batch Upload">
|
|
Process multiple documents with rate limiting:
|
|
|
|
```typescript
|
|
async function batchUpload(documents: Array<{id: string, content: string}>) {
|
|
const results = [];
|
|
|
|
for (const doc of documents) {
|
|
try {
|
|
const result = await client.add({
|
|
content: doc.content,
|
|
customId: doc.id,
|
|
containerTag: "batch_import"
|
|
});
|
|
results.push({ id: doc.id, success: true, docId: result.id });
|
|
} catch (error) {
|
|
results.push({ id: doc.id, success: false, error });
|
|
}
|
|
|
|
// Rate limit: 1 second between requests
|
|
await new Promise(r => setTimeout(r, 1000));
|
|
}
|
|
|
|
return results;
|
|
}
|
|
```
|
|
|
|
**Tips:**
|
|
- Batch size: 3-5 documents at once
|
|
- Delay: 1-2 seconds between requests
|
|
- Use `customId` to track and deduplicate
|
|
</Accordion>
|
|
|
|
<Accordion title="Error Handling">
|
|
| Status | Error | Cause |
|
|
|--------|-------|-------|
|
|
| 400 | BadRequestError | Missing required fields, invalid parameters |
|
|
| 401 | AuthenticationError | Invalid or missing API key |
|
|
| 403 | PermissionDeniedError | Insufficient permissions |
|
|
| 429 | RateLimitError | Too many requests or quota exceeded |
|
|
| 500 | InternalServerError | Processing failure |
|
|
|
|
```typescript
|
|
import { BadRequestError, RateLimitError } from 'supermemory';
|
|
|
|
try {
|
|
await client.add({ content: "..." });
|
|
} catch (error) {
|
|
if (error instanceof RateLimitError) {
|
|
// Wait and retry
|
|
await new Promise(r => setTimeout(r, 60000));
|
|
} else if (error instanceof BadRequestError) {
|
|
// Fix request parameters
|
|
console.error("Invalid request:", error.message);
|
|
}
|
|
}
|
|
```
|
|
</Accordion>
|
|
|
|
<Accordion title="Delete Content">
|
|
**Single delete:**
|
|
```typescript
|
|
await client.documents.delete("doc_id_123");
|
|
```
|
|
|
|
**Bulk delete by IDs:**
|
|
```typescript
|
|
await client.documents.deleteBulk({
|
|
ids: ["doc_1", "doc_2", "doc_3"]
|
|
});
|
|
```
|
|
|
|
**Bulk delete by container tag:**
|
|
```typescript
|
|
// Delete all content for a user
|
|
await client.documents.deleteBulk({
|
|
containerTags: ["user_123"]
|
|
});
|
|
```
|
|
|
|
Deletes are permanent — no recovery.
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
- [How to backfill historical data](/ingestion/batch-ingest-historical-data) — Import dated content with the batch API
|
|
- [Search Memories](/recall/search) — Query your content
|
|
- [User Profiles](/recall/user-profiles) — Get user context
|
|
- [Organizing & Filtering](/concepts/filtering) — Container tags and metadata
|