mirror of
https://github.com/lfnovo/open-notebook.git
synced 2026-05-01 21:00:43 +00:00
docs: restructure documentation with new organized layout
- Replace old docs structure with new comprehensive documentation - Organize into 8 major sections (0-START-HERE through 7-DEVELOPMENT) - Convert CONFIGURATION.md, CONTRIBUTING.md, MAINTAINER_GUIDE.md to redirects - Remove outdated MIGRATION.md and DESIGN_PRINCIPLES.md - Fix all internal documentation links and cross-references - Add progressive disclosure paths for different user types - Include 44 focused guides covering all features - Update README.md to remove v1.0 breaking changes notice
This commit is contained in:
LUIS NOVO
parent
71b8d13b24
commit
e13e4a2d8b
108 changed files with 16392 additions and 18153 deletions
475
docs/3-USER-GUIDE/search.md
Normal file
475
docs/3-USER-GUIDE/search.md
Normal file
|
|
@ -0,0 +1,475 @@
|
|||
# Search Effectively - Finding What You Need
|
||||
|
||||
Search is your gateway into your research. This guide covers two search modes and when to use each.
|
||||
|
||||
---
|
||||
|
||||
## Quick-Start: Find Something
|
||||
|
||||
### Simple Search
|
||||
|
||||
```
|
||||
1. Go to your notebook
|
||||
2. Type in search box
|
||||
3. See results (both sources and notes)
|
||||
4. Click result to view source/note
|
||||
5. Done!
|
||||
|
||||
That works for basic searches.
|
||||
But you can do much better...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Two Search Modes Explained
|
||||
|
||||
Open Notebook has two fundamentally different search approaches.
|
||||
|
||||
### Search Type 1: TEXT SEARCH (Keyword Matching)
|
||||
|
||||
**How it works:**
|
||||
- You search for words: "transformer"
|
||||
- System finds chunks containing "transformer"
|
||||
- Ranked by relevance: frequency, position, context
|
||||
|
||||
**Speed:** Very fast (instant)
|
||||
|
||||
**When to use:**
|
||||
- You remember exact words or phrases
|
||||
- You're looking for specific terms
|
||||
- You want precise keyword matches
|
||||
- You need exact quotes
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Search: "attention mechanism"
|
||||
Results:
|
||||
1. "The attention mechanism allows..." (perfect match)
|
||||
2. "Attention and other mechanisms..." (partial match)
|
||||
3. "How mechanisms work in attention..." (includes words separately)
|
||||
|
||||
All contain "attention" AND "mechanism"
|
||||
Ranked by how close together they are
|
||||
```
|
||||
|
||||
**What it finds:**
|
||||
- Exact phrases: "transformer model"
|
||||
- Individual words: transformer OR model (too broad)
|
||||
- Names: "Vaswani et al."
|
||||
- Numbers: "1994", "GPT-4"
|
||||
- Technical terms: "LSTM", "convolution"
|
||||
|
||||
**What it doesn't find:**
|
||||
- Similar words: searching "attention" won't find "focus"
|
||||
- Synonyms: searching "large" won't find "big"
|
||||
- Concepts: searching "similarity" won't find "likeness"
|
||||
|
||||
---
|
||||
|
||||
### Search Type 2: VECTOR SEARCH (Semantic/Concept Matching)
|
||||
|
||||
**How it works:**
|
||||
- Your search converted to embedding (vector)
|
||||
- All chunks converted to embeddings
|
||||
- System finds most similar embeddings
|
||||
- Ranked by semantic similarity
|
||||
|
||||
**Speed:** A bit slower (1-2 seconds)
|
||||
|
||||
**When to use:**
|
||||
- You're exploring a concept
|
||||
- You don't know exact words
|
||||
- You want semantically similar content
|
||||
- You're discovering, not searching
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Search: "What's the mechanism for understanding in models?"
|
||||
(Notice: No chunk likely says exactly that)
|
||||
|
||||
Results:
|
||||
1. "Mechanistic interpretability allows understanding..." (semantic match)
|
||||
2. "Feature attribution reveals how models work..." (conceptually similar)
|
||||
3. "Attention visualization shows model decisions..." (same topic)
|
||||
|
||||
None contain your exact words
|
||||
But all are semantically related
|
||||
```
|
||||
|
||||
**What it finds:**
|
||||
- Similar concepts: "understanding" + "interpretation" + "explainability" (all related)
|
||||
- Paraphrases: "big" and "large" (same meaning)
|
||||
- Related ideas: "safety" relates to "alignment" (connected concepts)
|
||||
- Analogies: content about biological learning when searching "learning"
|
||||
|
||||
**What it doesn't find:**
|
||||
- Exact keywords: if you search a rare word, vector search might miss it
|
||||
- Specific numbers: "1994" vs "1993" are semantically different
|
||||
- Technical jargon: "LSTM" and "RNN" are different even if related
|
||||
|
||||
---
|
||||
|
||||
## Decision: Text Search vs. Vector Search?
|
||||
|
||||
```
|
||||
Question: "Do I remember the exact words?"
|
||||
|
||||
→ YES: Use TEXT SEARCH
|
||||
Example: "I remember the paper said 'attention is all you need'"
|
||||
|
||||
→ NO: Use VECTOR SEARCH
|
||||
Example: "I'm looking for content about how models process information"
|
||||
|
||||
→ UNSURE: Try TEXT SEARCH first (faster)
|
||||
If no results, try VECTOR SEARCH
|
||||
|
||||
Text search: "I know what I'm looking for"
|
||||
Vector search: "I'm exploring an idea"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step-by-Step: Using Each Search
|
||||
|
||||
### Text Search
|
||||
|
||||
```
|
||||
1. Go to search box
|
||||
2. Type your keywords: "transformer", "attention", "2017"
|
||||
3. Press Enter
|
||||
4. Results appear (usually instant)
|
||||
5. Click result to see context
|
||||
|
||||
Results show:
|
||||
- Which source contains it
|
||||
- How many times it appears
|
||||
- Relevance score
|
||||
- Preview of surrounding text
|
||||
```
|
||||
|
||||
### Vector Search
|
||||
|
||||
```
|
||||
1. Go to search box
|
||||
2. Type your concept: "How do models understand language?"
|
||||
3. Choose "Vector Search" from dropdown
|
||||
4. Press Enter
|
||||
5. Results appear (1-2 seconds)
|
||||
6. Click result to see context
|
||||
|
||||
Results show:
|
||||
- Semantically related chunks
|
||||
- Similarity score (higher = more related)
|
||||
- Preview of surrounding text
|
||||
- Different sources mixed together
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The Ask Feature (Automated Search)
|
||||
|
||||
Ask is different from simple search. It automatically searches, synthesizes, and answers.
|
||||
|
||||
### How Ask Works
|
||||
|
||||
```
|
||||
Stage 1: QUESTION UNDERSTANDING
|
||||
"Compare the approaches in my papers"
|
||||
→ System: "This asks for comparison"
|
||||
|
||||
Stage 2: SEARCH STRATEGY
|
||||
→ System: "I should search for each approach separately"
|
||||
|
||||
Stage 3: PARALLEL SEARCHES
|
||||
→ Search 1: "Approach in paper A"
|
||||
→ Search 2: "Approach in paper B"
|
||||
(Multiple searches happen at once)
|
||||
|
||||
Stage 4: ANALYSIS & SYNTHESIS
|
||||
→ Per-result analysis: "Based on paper A, the approach is..."
|
||||
→ Per-result analysis: "Based on paper B, the approach is..."
|
||||
→ Final synthesis: "Comparing A and B: A differs from B in..."
|
||||
|
||||
Result: Comprehensive answer, not just search results
|
||||
```
|
||||
|
||||
### When to Use Ask vs. Simple Search
|
||||
|
||||
| Task | Use | Why |
|
||||
|------|-----|-----|
|
||||
| "Find the quote about X" | **TEXT SEARCH** | Need exact words |
|
||||
| "What does source A say about X?" | **TEXT SEARCH** | Direct, fast answer |
|
||||
| "Find content about X" | **VECTOR SEARCH** | Semantic discovery |
|
||||
| "Compare A and B" | **ASK** | Comprehensive synthesis |
|
||||
| "What's the big picture?" | **ASK** | Full analysis needed |
|
||||
| "How do these sources relate?" | **ASK** | Cross-source synthesis |
|
||||
| "I remember something about X" | **TEXT SEARCH** | Recall memory |
|
||||
| "I'm exploring the topic of X" | **VECTOR SEARCH** | Discovery mode |
|
||||
|
||||
---
|
||||
|
||||
## Advanced Search Strategies
|
||||
|
||||
### Strategy 1: Simple Search with Follow-Up
|
||||
|
||||
```
|
||||
1. Text search: "attention mechanism"
|
||||
Results: 50 matches
|
||||
|
||||
2. Too many. Follow up with vector search:
|
||||
"Why is attention useful?" (concept search)
|
||||
Results: Most relevant papers/notes
|
||||
|
||||
3. Better results with less noise
|
||||
```
|
||||
|
||||
### Strategy 2: Ask for Comprehensive, Then Search for Details
|
||||
|
||||
```
|
||||
1. Ask: "What are the main approaches to X?"
|
||||
Result: Comprehensive answer about A, B, C
|
||||
|
||||
2. Use that to identify specific sources
|
||||
|
||||
3. Text search in those specific sources:
|
||||
"Why did they choose method X?"
|
||||
Result: Detailed information
|
||||
```
|
||||
|
||||
### Strategy 3: Vector Search for Discovery, Text for Verification
|
||||
|
||||
```
|
||||
1. Vector search: "How do transformers generalize?"
|
||||
Results: Related conceptual papers
|
||||
|
||||
2. Skim to understand landscape
|
||||
|
||||
3. Text search in promising sources:
|
||||
"generalization", "extrapolation", "transfer"
|
||||
Results: Specific passages to read carefully
|
||||
```
|
||||
|
||||
### Strategy 4: Combine Search with Chat
|
||||
|
||||
```
|
||||
1. Vector search: "What's new in AI 2026?"
|
||||
Results: Latest papers
|
||||
|
||||
2. Go to Chat
|
||||
3. Add those papers to context
|
||||
4. Ask detailed follow-up questions
|
||||
5. Get deep analysis of results
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Search Quality Issues & Fixes
|
||||
|
||||
### Getting No Results
|
||||
|
||||
| Problem | Cause | Solution |
|
||||
|---------|-------|----------|
|
||||
| Text search: no results | Word doesn't appear | Try vector search instead |
|
||||
| Vector search: no results | Concept not in content | Try broader search term |
|
||||
| Both empty | Content not in notebook | Add sources to notebook |
|
||||
| | Sources not processed | Wait for processing to complete |
|
||||
|
||||
### Getting Too Many Results
|
||||
|
||||
| Problem | Cause | Solution |
|
||||
|---------|-------|----------|
|
||||
| 1000+ results | Search too broad | Be more specific |
|
||||
| | All sources | Filter by source |
|
||||
| | Keyword matches rare words | Use vector search instead |
|
||||
|
||||
### Getting Wrong Results
|
||||
|
||||
| Problem | Cause | Solution |
|
||||
|---------|-------|----------|
|
||||
| Results irrelevant | Search term has multiple meanings | Provide more context |
|
||||
| | Using text search for concepts | Try vector search |
|
||||
| Different meaning | Homonym (word means multiple things) | Add context (e.g., "attention mechanism") |
|
||||
|
||||
### Getting Low Quality Results
|
||||
|
||||
| Problem | Cause | Solution |
|
||||
|---------|-------|----------|
|
||||
| Results don't match intent | Vague search term | Be specific ("Who invented X?" vs "X") |
|
||||
| | Concept not well-represented | Add more sources on that topic |
|
||||
| | Vector embedding not trained on domain | Use text search as fallback |
|
||||
|
||||
---
|
||||
|
||||
## Tips for Better Searches
|
||||
|
||||
### For Text Search
|
||||
1. **Be specific** — "attention mechanism" not just "attention"
|
||||
2. **Use exact phrases** — Put quotes around: "attention is all you need"
|
||||
3. **Include context** — "LSTM vs attention" not just "attention"
|
||||
4. **Use technical terms** — These are usually more precise
|
||||
5. **Try synonyms** — If first search fails, try related terms
|
||||
|
||||
### For Vector Search
|
||||
1. **Ask a question** — "What's the best way to X?" is better than "best way"
|
||||
2. **Use natural language** — Explain what you're looking for
|
||||
3. **Be specific about intent** — "Compare X and Y" not "X and Y"
|
||||
4. **Include context** — "In machine learning, how..." vs just "how..."
|
||||
5. **Think conceptually** — What idea are you exploring?
|
||||
|
||||
### General Tips
|
||||
1. **Start broad, then narrow** — "AI papers" → "transformers" → "attention mechanism"
|
||||
2. **Try both search types** — Each finds different things
|
||||
3. **Use Ask for complex questions** — Don't just search
|
||||
4. **Save good results as notes** — Create knowledge base
|
||||
5. **Filter by source if needed** — "Search in Paper A only"
|
||||
|
||||
---
|
||||
|
||||
## Search Examples
|
||||
|
||||
### Example 1: Finding a Specific Fact
|
||||
|
||||
**Goal:** "Find the date the transformer was introduced"
|
||||
|
||||
```
|
||||
Step 1: Text search
|
||||
"transformer 2017" (or year you remember)
|
||||
|
||||
If that works: Done!
|
||||
|
||||
If no results: Try
|
||||
"attention is all you need" (famous paper title)
|
||||
|
||||
Check result for exact date
|
||||
```
|
||||
|
||||
### Example 2: Exploring a Concept
|
||||
|
||||
**Goal:** "Find content about alignment interpretability"
|
||||
|
||||
```
|
||||
Step 1: Vector search
|
||||
"How do we make AI interpretable?"
|
||||
|
||||
Results: Papers on interpretability, transparency, alignment
|
||||
|
||||
Step 2: Review results
|
||||
See which papers are most relevant
|
||||
|
||||
Step 3: Deep dive
|
||||
Go to Chat, add top 2-3 papers
|
||||
Ask detailed questions about alignment
|
||||
```
|
||||
|
||||
### Example 3: Comprehensive Answer
|
||||
|
||||
**Goal:** "How do different approaches to AI safety compare?"
|
||||
|
||||
```
|
||||
Step 1: Ask
|
||||
"Compare the main approaches to AI safety in my sources"
|
||||
|
||||
Result: Comprehensive analysis comparing approaches
|
||||
|
||||
Step 2: Identify sources
|
||||
From answer, see which papers were most relevant
|
||||
|
||||
Step 3: Deep dive
|
||||
Text search in those papers:
|
||||
"limitations", "critiques", "open problems"
|
||||
|
||||
Step 4: Save as notes
|
||||
Create comparison note from Ask result
|
||||
```
|
||||
|
||||
### Example 4: Finding Pattern
|
||||
|
||||
**Goal:** "Find all papers mentioning transformers"
|
||||
|
||||
```
|
||||
Step 1: Text search
|
||||
"transformer"
|
||||
|
||||
Results: All papers mentioning "transformer"
|
||||
|
||||
Step 2: Vector search
|
||||
"neural network architecture for sequence processing"
|
||||
|
||||
Results: Papers that don't say "transformer" but discuss similar concept
|
||||
|
||||
Step 3: Combine
|
||||
Union of text + vector results shows full landscape
|
||||
|
||||
Step 4: Analyze
|
||||
Go to Chat with all results
|
||||
Ask: "What's common across all these?"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Search in the Workflow
|
||||
|
||||
How search fits with other features:
|
||||
|
||||
```
|
||||
SOURCES
|
||||
↓
|
||||
SEARCH (find what matters)
|
||||
├─ Text search (precise)
|
||||
├─ Vector search (exploration)
|
||||
└─ Ask (comprehensive)
|
||||
↓
|
||||
CHAT (explore with follow-ups)
|
||||
↓
|
||||
TRANSFORMATIONS (batch extract)
|
||||
↓
|
||||
NOTES (save insights)
|
||||
```
|
||||
|
||||
### Workflow Example
|
||||
|
||||
```
|
||||
1. Add 10 papers to notebook
|
||||
|
||||
2. Search: "What's the state of the art?"
|
||||
(Vector search explores landscape)
|
||||
|
||||
3. Ask: "Compare these 3 approaches"
|
||||
(Comprehensive synthesis)
|
||||
|
||||
4. Chat: Deep questions about winner
|
||||
(Follow-up exploration)
|
||||
|
||||
5. Save best insights as notes
|
||||
(Knowledge capture)
|
||||
|
||||
6. Transform remaining papers
|
||||
(Batch extraction for later)
|
||||
|
||||
7. Create podcast from notes + sources
|
||||
(Share findings)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary: Know Your Search
|
||||
|
||||
**TEXT SEARCH** — "I know what I'm looking for"
|
||||
- Fast, precise, keyword-based
|
||||
- Use when you remember exact words/phrases
|
||||
- Best for: Finding specific facts, quotes, technical terms
|
||||
- Speed: Instant
|
||||
|
||||
**VECTOR SEARCH** — "I'm exploring an idea"
|
||||
- Slow-ish, concept-based, semantic
|
||||
- Use when you're discovering connections
|
||||
- Best for: Concept exploration, related ideas, synonyms
|
||||
- Speed: 1-2 seconds
|
||||
|
||||
**ASK** — "I want a comprehensive answer"
|
||||
- Auto-searches, auto-analyzes, synthesizes
|
||||
- Use for complex questions needing multiple sources
|
||||
- Best for: Comparisons, big-picture questions, synthesis
|
||||
- Speed: 10-30 seconds
|
||||
|
||||
Pick the right tool for your search goal, and you'll find what you need faster.
|
||||
Loading…
Add table
Add a link
Reference in a new issue