vrr/open-notebook
2
0
Fork 0
mirror of https://github.com/lfnovo/open-notebook.git synced 2026-05-01 21:00:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs: update quick start guides and core concepts for clarity and accuracy

Browse source
This commit is contained in:
LUIS NOVO 2026-01-04 09:09:20 -03:00
parent e13e4a2d8b
commit 655d164fa0
13 changed files with 220 additions and 169 deletions
Download patch file Download diff file Expand all files Collapse all files

123
docs/2-CORE-CONCEPTS/ai-context-rag.md
View file

@ -1,6 +1,6 @@
# AI Context & RAG - How Open Notebook Uses Your Research
The core innovation in Open Notebook is how it makes AI models aware of your private research without uploading everything to the cloud. This section explains the "why" and "how" of that system.
Open Notebook uses different approaches to make AI models aware of your research depending on the feature. This section explains **RAG** (used in Ask) and **full-content context** (used in Chat).
---
@ -23,10 +23,17 @@ The core innovation in Open Notebook is how it makes AI models aware of your pri
- Pro: Private, free
- Con: AI doesn't know anything about your specific topic
### Open Notebook's Solution: RAG
**RAG** = Retrieval-Augmented Generation
### Open Notebook's Dual Approach
The insight: *Instead of changing the model, change what information you feed it.*
**For Chat**: Sends the entire selected content to the LLM
- Simple and transparent: You select sources, they're sent in full
- Maximum context: AI sees everything you choose
- You control which sources are included
**For Ask (RAG)**: Retrieval-Augmented Generation
- RAG = Retrieval-Augmented Generation
- The insight: *Search your content, find relevant pieces, send only those*
- Automatic: AI decides what's relevant based on your question
---
@ -260,45 +267,83 @@ Result: AI doesn't get distracted; gives better answer
## The Difference: Chat vs. Ask
Both use RAG, but differently.
**IMPORTANT**: These use completely different approaches!
### Chat: Manual Context Control
### Chat: Full-Content Context (NO RAG)
**How it works:**
```
YOU:
1. Choose which sources to include
1. Select which sources to include in context
2. Set context level (full/summary/excluded)
3. Ask question
SYSTEM:
- Uses ONLY the sources you selected
- Respects your context levels
- Answers based on what you chose
- Takes ALL selected sources (respecting context levels)
- Sends the ENTIRE content to the LLM at once
- NO search, NO retrieval, NO chunking
- AI sees everything you selected
YOU:
4. Ask follow-up (context stays the same)
5. Or change context for next question
AI:
- Responds based on the full content you provided
- Can reference any part of selected sources
- Conversational: context stays for follow-ups
```
**Use this when**: You know which sources matter for THIS conversation.
**Use this when**:
- You know which sources are relevant
- You want conversational back-and-forth
- You want AI to see the complete context
- You're doing close reading or analysis
### Ask: Automatic Search
**Advantages:**
- Simple and transparent
- AI sees everything (no missed content)
- Conversational flow
**Limitations:**
- Limited by LLM context window
- You must manually select relevant sources
- Sends more tokens (higher cost with many sources)
---
### Ask: RAG - Automatic Retrieval
**How it works:**
```
YOU:
Ask one complex question
SYSTEM:
1. Analyzes your question (using smart model)
2. Breaks it into searchable parts
3. Automatically searches your sources
4. Retrieves relevant chunks
5. Processes results
1. Analyzes your question
2. Searches across ALL your sources automatically
3. Finds relevant chunks using vector similarity
4. Retrieves only the most relevant pieces
5. Sends ONLY those chunks to the LLM
6. Synthesizes into comprehensive answer
YOU:
Get one detailed answer (not conversational)
AI:
- Sees ONLY the retrieved chunks (not full sources)
- Answers based on what was found to be relevant
- One-shot answer (not conversational)
```
**Use this when**: You want a comprehensive answer and trust the AI to find what's relevant.
**Use this when**:
- You have many sources and don't know which are relevant
- You want the AI to search automatically
- You need a comprehensive answer to a complex question
- You want to minimize tokens sent to LLM
**Advantages:**
- Automatic search (you don't pick sources)
- Works across many sources at once
- Cost-effective (sends only relevant chunks)
**Limitations:**
- Not conversational (single question/answer)
- AI only sees retrieved chunks (might miss context)
- Search quality depends on how well question matches content
---
@ -380,16 +425,26 @@ This is why semantic search finds conceptually similar content even when words a
## Summary
**RAG** allows Open Notebook to:
Open Notebook gives you **two ways** to work with AI:
### Chat (Full-Content)
- Sends entire selected sources to LLM
- Manual control: you pick sources
- Conversational: back-and-forth dialog
- Transparent: you know exactly what AI sees
- Best for: focused analysis, close reading
### Ask (RAG)
- Searches and retrieves relevant chunks automatically
- Automatic: AI finds what's relevant
- One-shot: single comprehensive answer
- Efficient: sends only relevant pieces
- Best for: broad questions across many sources
**Both approaches:**
1. Keep your data private (doesn't leave your system by default)
2. Make AI aware of your research (retrieval brings in relevant chunks)
3. Give you control (you decide what's in context)
4. Create audit trails (citations show what was used)
5. Support multiple searches (keyword and semantic)
2. Give you control (you choose which features to use)
3. Create audit trails (citations show what was used)
4. Support multiple AI providers
This is fundamentally different from:
- Systems that fine-tune (slow, permanent)
- Systems that send everything (privacy nightmare)
- Systems that ignore your data (no customization)
It's **retrieval-augmented generation**: the system retrieves what's relevant, then augments the AI's knowledge with it.
**Coming Soon**: The community is working on adding RAG capabilities to Chat as well, giving you the best of both worlds.

73
docs/2-CORE-CONCEPTS/chat-vs-transformations.md
View file

@ -93,30 +93,32 @@ You: [Get back one comprehensive answer]
---
### 3. TRANSFORMATIONS - Batch Processing with Templates
### 3. TRANSFORMATIONS - Template-Based Processing
**What it is:** Apply a reusable template to one or more sources and get structured outputs.
**What it is:** Apply a reusable template to a source and get structured output.
**The flow:**
```
1. You define a transformation (or choose a preset)
"Extract: main argument, methodology, limitations"
2. You apply it to one or more sources
OR system applies it to all sources
2. You apply it to ONE source at a time
(You can repeat for other sources)
3. For each source:
3. For the source:
- Source content + transformation prompt → AI
- Result stored as new insight/note
4. You get back
- Structured outputs (main argument, methodology, limitations)
- Saved as notes in your notebook
- Structured output (main argument, methodology, limitations)
- Saved as a note in your notebook
```
**Context management:** You choose which sources to transform.
**Context management:** Works on one source at a time.
**Batch processing:** Process multiple sources at once.
**Reusable:** Apply the same template to different sources (one by one).
**Note**: Currently processes one source at a time. Batch processing (multiple sources at once) is planned for a future release.
**Example:**
```
@ -128,23 +130,24 @@ You: Define transformation
- Limitations and gaps
- Recommended next research"
You: Apply to 10 papers
You: Apply to paper 1
System:
- For each paper, runs the transformation
- Results stored as 10 new notes
- Each note has the structure you defined
- Runs the transformation on paper 1
- Result stored as new note
You: Now you have structured notes from all 10 papers
Perfect for writing a literature review or comparison
You: Apply same transformation to paper 2, 3, etc.
After 10 papers:
- You have 10 structured notes with consistent format
- Perfect for writing a literature review or comparison
```
**Best for:**
- Extracting the same information from multiple sources
- Creating structured summaries
- Extracting the same information from each source (run repeatedly)
- Creating structured summaries with consistent format
- Building a knowledge base of categorized insights
- When you want consistent, reusable templates
- Batch processing multiple sources
- When you want reusable templates you can apply to each source
---
@ -165,8 +168,8 @@ What are you trying to do?
├─→ "I need to compare these sources or get a comprehensive answer"
│ └─→ USE: ASK
│
├─→ "I want to extract the same info from many sources"
│ └─→ USE: TRANSFORMATIONS
├─→ "I want to extract the same info from each source (one at a time)"
│ └─→ USE: TRANSFORMATIONS (apply to each source)
│
└─→ "I just want to read and search"
└─→ USE: Search (text or vector)
@ -179,13 +182,13 @@ What are you trying to do?
| Aspect | CHAT | ASK | TRANSFORMATIONS |
|--------|------|-----|-----------------|
| **What's it for?** | Conversational exploration | Comprehensive Q&A | Batch extraction |
| **# of questions** | Multiple (conversational) | One | One template, many sources |
| **Context control** | Manual (you choose) | Automatic (system searches) | Manual (you choose) |
| **Conversational?** | Yes (follow-ups work) | No (one question only) | No (batch process) |
| **Output** | Natural conversation | Natural answer | Structured notes |
| **Time** | Quick (back-and-forth) | Longer (comprehensive) | Batch (all at once) |
| **Best when** | Exploring & uncertain | Need full picture | Want consistency |
| **What's it for?** | Conversational exploration | Comprehensive Q&A | Template-based extraction |
| **# of questions** | Multiple (conversational) | One | One template per source |
| **Context control** | Manual (you choose) | Automatic (system searches) | One source at a time |
| **Conversational?** | Yes (follow-ups work) | No (one question only) | No (single operation) |
| **Output** | Natural conversation | Natural answer | Structured note |
| **Time** | Quick (back-and-forth) | Longer (comprehensive) | Per source |
| **Best when** | Exploring & uncertain | Need full picture | Want consistent format |
| **Model speed** | Any | Fast preferred | Any |
---
@ -199,8 +202,10 @@ Goal: Write literature review from 15 papers
Step 1: TRANSFORMATIONS
- Define: "Extract abstract, methodology, findings, relevance"
- Apply to all 15 papers
- Get back 15 structured notes
- Apply to paper 1 → get structured note
- Apply to paper 2 → get structured note
- ... repeat for all 15 papers
- Result: 15 structured notes with consistent format
Step 2: Read the notes
- Now you have consistent summaries
@ -231,7 +236,7 @@ Step 3: CHAT
Step 4: TRANSFORMATIONS (optional)
- Define: "Extract: pain point, frequency, who mentioned it"
- Apply to all interviews
- Apply to each interview (one by one)
- Get structured data for analysis
```
@ -338,11 +343,11 @@ CHAT → Save as Note → TRANSFORMATIONS
|-----------|-----|-----|
| "I want to explore a topic with follow-up questions" | **CHAT** | Conversational, you control context |
| "I need a comprehensive answer to one complex question" | **ASK** | Automatic search, synthesized answer |
| "I want consistent summaries from 10+ sources" | **TRANSFORMATIONS** | Template reuse, batch processing |
| "I want consistent summaries from each source" | **TRANSFORMATIONS** | Template reuse, apply to each source |
| "I'm comparing two specific sources" | **CHAT** | Select just those 2, have discussion |
| "I need to categorize all sources by X criteria" | **TRANSFORMATIONS** | Extract category for each source |
| "I need to categorize each source by X criteria" | **TRANSFORMATIONS** | Extract category from each source |
| "I want to understand the big picture across all sources" | **ASK** | Automatic comprehensive search |
| "I want to build a knowledge base" | **TRANSFORMATIONS** | Create structured notes |
| "I want to build a knowledge base" | **TRANSFORMATIONS** | Create structured note from each source |
| "I want to iterate on understanding" | **CHAT** | Multiple questions, refine thinking |
The key insight: **Different questions need different tools.** Open Notebook gives you all three because research rarely fits one mode.

10
docs/2-CORE-CONCEPTS/index.md
View file

@ -12,16 +12,16 @@ How Open Notebook organizes your research. Understand the three-tier container s
---
### 2. [AI Context & RAG](ai-context-rag.md)
How Open Notebook makes AI aware of your research without uploading everything to the cloud.
How Open Notebook makes AI aware of your research - two different approaches.
**Key idea**: RAG (Retrieval-Augmented Generation) means the AI searches your content, finds relevant pieces, and answers based on what it found. You control which content is in scope.
**Key idea**: **Chat** sends entire selected sources to the LLM (full context, conversational). **Ask** uses RAG (retrieval-augmented generation) to automatically search and retrieve only relevant chunks. Different tools for different needs.
---
### 3. [Chat vs. Transformations](chat-vs-transformations.md)
Why Open Notebook has different interaction modes and when to use each one.
**Key idea**: Chat is conversational exploration (you control context). Transformations are batch processing (you define the template). They answer different questions.
**Key idea**: Chat is conversational exploration (you control context). Transformations are insight extractions. They reduced content to smaller bits of concentrated/dense information, which is much more suitable for an AI to use.
---
@ -42,8 +42,8 @@ Why Open Notebook can turn research into audio and why this matters.
## Read This Section If:
- **You're new to Open Notebook** — Start here to understand how the system works conceptually before learning the features
- **You're confused about RAG** — Section 2 explains what it is and why it matters
- **You're wondering when to use Chat vs Ask** — Section 3 clarifies the differences
- **You're confused about Chat vs Ask** — Section 2 explains the difference (full-content vs RAG)
- **You're wondering when to use Chat vs Transformations** — Section 3 clarifies the differences
- **You want to understand privacy controls** — Section 4 shows you what you can control
- **You're curious about podcasts** — Section 5 explains the architecture and why it's different from competitors