docs: new docs

This commit is contained in:
LUIS NOVO 2025-07-17 12:38:40 -03:00
parent 3bb691d0b8
commit b20c62df47
44 changed files with 12929 additions and 1853 deletions

View file

@ -0,0 +1,845 @@
# Open Notebook Installation Guide
This comprehensive guide will help you install and configure Open Notebook, an open-source, privacy-focused alternative to Google's Notebook LM. Whether you're a beginner or advanced user, this guide covers all installation methods and configuration options.
## Table of Contents
1. [Quick Start](#quick-start)
2. [System Requirements](#system-requirements)
3. [Installation Methods](#installation-methods)
4. [Service Architecture](#service-architecture)
5. [Environment Configuration](#environment-configuration)
6. [Manual Installation](#manual-installation)
7. [Docker Installation](#docker-installation)
8. [AI Model Configuration](#ai-model-configuration)
9. [Verification and Testing](#verification-and-testing)
10. [Security Configuration](#security-configuration)
11. [Troubleshooting](#troubleshooting)
---
## Quick Start
For users who want to get started immediately:
### Docker (Recommended for Beginners)
```bash
# Create project directory
mkdir open-notebook && cd open-notebook
# Download configuration files
curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/.env.example
# Rename and configure environment
mv .env.example docker.env
# Edit docker.env with your API keys
# Start Open Notebook
docker compose up -d
```
### From Source (Developers)
```bash
# Clone and setup
git clone https://github.com/lfnovo/open-notebook
cd open-notebook
cp .env.example .env
# Edit .env with your API keys
# Install dependencies and start
uv sync
make start-all
```
Access Open Notebook at `http://localhost:8502`
---
## System Requirements
### Hardware Requirements
- **CPU**: 2+ cores recommended (4+ cores for better performance)
- **RAM**: Minimum 4GB (8GB+ recommended)
- **Storage**: 10GB+ available space
- **Network**: Stable internet connection for AI model access
### Operating System Support
- **macOS**: 10.15 (Catalina) or later
- **Linux**: Ubuntu 18.04+, Debian 9+, CentOS 7+, Fedora 30+
- **Windows**: Windows 10 or later (WSL2 recommended)
### Software Prerequisites
- **Python**: 3.9 or later (for source installation)
- **Docker**: Latest version (for Docker installation)
- **uv**: Python package manager (for source installation)
---
## Installation Methods
Open Notebook supports multiple installation methods. Choose the one that best fits your needs:
| Method | Best For | Difficulty | Pros | Cons |
|--------|----------|------------|------|------|
| **Docker Single-Container** | Beginners, simple deployments | Easy | One-click setup, isolated environment | Less control, harder to debug |
| **Docker Multi-Container** | Production deployments | Medium | Scalable, professional setup | More complex configuration |
| **Source Installation** | Developers, customization | Advanced | Full control, easy debugging | Requires Python knowledge |
---
## Service Architecture
Open Notebook consists of four main services that work together:
### 1. **SurrealDB Database** (Port 8000)
- **Purpose**: Stores notebooks, sources, notes, and metadata
- **Technology**: SurrealDB - a modern, multi-model database
- **Configuration**: Runs in Docker container with persistent storage
### 2. **FastAPI Backend** (Port 5055)
- **Purpose**: REST API for all application functionality
- **Features**: Interactive API documentation, authentication, data validation
- **Endpoints**: `/api/notebooks`, `/api/sources`, `/api/notes`, `/api/chat`
### 3. **Background Worker**
- **Purpose**: Processes long-running tasks asynchronously
- **Tasks**: Podcast generation, content transformations, embeddings
- **Technology**: Surreal Commands worker system
### 4. **Streamlit UI** (Port 8502)
- **Purpose**: Web-based user interface
- **Features**: Notebooks, chat, sources, notes, search
- **Technology**: Streamlit framework
### Service Communication Flow
```
User Browser → Streamlit UI → FastAPI Backend → SurrealDB Database
Background Worker ← Job Queue
```
---
## Environment Configuration
Open Notebook uses environment variables for configuration. Create a `.env` file (or `docker.env` for Docker) based on the template below:
### Core Configuration
```env
# Security (Optional - for public deployments)
OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
# Database Configuration
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="production"
```
### AI Provider Configuration
#### OpenAI (Recommended for beginners)
```env
# Provides: Language models, embeddings, TTS, STT
OPENAI_API_KEY=sk-your-openai-key-here
```
#### Anthropic (Claude models)
```env
# Provides: High-quality language models
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key-here
```
#### Google (Gemini)
```env
# Provides: Large context models, embeddings, TTS
GEMINI_API_KEY=your-gemini-key-here
```
#### Vertex AI (Google Cloud)
```env
# Provides: Enterprise-grade AI models
VERTEX_PROJECT=your-google-cloud-project-name
GOOGLE_APPLICATION_CREDENTIALS=./google-credentials.json
VERTEX_LOCATION=us-east5
```
#### Additional Providers
```env
# DeepSeek - Cost-effective models
DEEPSEEK_API_KEY=your-deepseek-key-here
# Mistral - European AI provider
MISTRAL_API_KEY=your-mistral-key-here
# Groq - Fast inference
GROQ_API_KEY=your-groq-key-here
# xAI (Grok) - Cutting-edge models
XAI_API_KEY=your-xai-key-here
# ElevenLabs - High-quality voice synthesis
ELEVENLABS_API_KEY=your-elevenlabs-key-here
# Ollama - Local AI models
OLLAMA_API_BASE="http://localhost:11434"
# OpenRouter - Access to multiple models
OPENROUTER_BASE_URL="https://openrouter.ai/api/v1"
OPENROUTER_API_KEY=your-openrouter-key-here
# Azure OpenAI
AZURE_OPENAI_API_KEY=your-azure-key-here
AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
AZURE_OPENAI_API_VERSION="2024-12-01-preview"
AZURE_OPENAI_DEPLOYMENT_NAME=your-deployment-name
```
### Optional Services
```env
# Firecrawl - Enhanced web scraping
FIRECRAWL_API_KEY=your-firecrawl-key-here
# Jina - Advanced embeddings
JINA_API_KEY=your-jina-key-here
# Voyage AI - Specialized embeddings
VOYAGE_API_KEY=your-voyage-key-here
# LangSmith - Debugging and monitoring
LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
LANGCHAIN_API_KEY=your-langsmith-key-here
LANGCHAIN_PROJECT="Open Notebook"
```
---
## Manual Installation
### Prerequisites Installation
#### macOS
```bash
# Install Homebrew if not already installed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Install system dependencies
brew install libmagic
# Install uv (Python package manager)
brew install uv
# Install Docker Desktop
brew install --cask docker
```
#### Ubuntu/Debian
```bash
# Update package list
sudo apt update
# Install system dependencies
sudo apt install -y libmagic-dev python3-dev build-essential
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install Docker
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```
#### CentOS/RHEL/Fedora
```bash
# Install system dependencies
sudo dnf install -y file-devel python3-devel gcc
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install Docker
sudo dnf install -y docker docker-compose
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```
### Source Installation Steps
1. **Clone the Repository**
```bash
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
```
2. **Configure Environment**
```bash
# Copy environment template
cp .env.example .env
# Edit environment file with your API keys
nano .env # or use your preferred editor
```
3. **Install Python Dependencies**
```bash
# Install all required packages
uv sync
# Install additional system-specific packages
uv pip install python-magic
```
4. **Initialize Database**
```bash
# Start SurrealDB
make database
# Wait for database to be ready (about 10 seconds)
```
5. **Start All Services**
```bash
# Start all services at once
make start-all
```
This will start:
- SurrealDB database on port 8000
- FastAPI backend on port 5055
- Background worker for processing
- Streamlit UI on port 8502
### Alternative: Start Services Individually
For development or debugging, you can start each service separately:
```bash
# Terminal 1: Database
make database
# Terminal 2: API Backend
make api
# Terminal 3: Background Worker
make worker
# Terminal 4: Streamlit UI
make run
```
---
## Docker Installation
### Single-Container Deployment (Recommended for Beginners)
Perfect for personal use or platforms like PikaPods:
1. **Create Project Directory**
```bash
mkdir open-notebook
cd open-notebook
```
2. **Create Docker Compose File**
```bash
# Create docker-compose.yml
cat > docker-compose.yml << 'EOF'
services:
open_notebook:
image: lfnovo/open_notebook:latest-single
ports:
- "8502:8502"
- "5055:5055"
env_file:
- ./docker.env
pull_policy: always
volumes:
- ./notebook_data:/app/data
- ./surreal_single_data:/mydata
restart: always
EOF
```
3. **Create Environment File**
```bash
# Create docker.env with your API keys
cat > docker.env << 'EOF'
# REQUIRED: At least one AI provider
OPENAI_API_KEY=your-openai-key-here
# Database settings (don't change)
SURREAL_ADDRESS=localhost
SURREAL_PORT=8000
SURREAL_USER=root
SURREAL_PASS=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
# Optional: Password protection
# OPEN_NOTEBOOK_PASSWORD=your_secure_password
EOF
```
4. **Start Open Notebook**
```bash
docker compose up -d
```
### Multi-Container Deployment (Production)
For scalable production deployments:
1. **Download Configuration**
```bash
# Download the main docker-compose.yml
curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
# Copy environment template
curl -o docker.env https://raw.githubusercontent.com/lfnovo/open-notebook/main/.env.example
```
2. **Configure Environment**
```bash
# Edit docker.env with your API keys
nano docker.env
```
3. **Start Services**
```bash
# Start with multi-container profile
docker compose --profile multi up -d
```
### Docker Service Management
```bash
# Check service status
docker compose ps
# View logs
docker compose logs -f
# Stop services
docker compose down
# Update to latest version
docker compose pull
docker compose up -d
# Restart specific service
docker compose restart open_notebook
```
---
## AI Model Configuration
After installation, configure your AI models for optimal performance:
### 1. Access Model Settings
- Navigate to **Settings****Models** in the web interface
- Or visit `http://localhost:8502` and click the settings icon
### 2. Configure Model Categories
#### Language Models (Chat & Generation)
**Budget-Friendly Options:**
- `gpt-4o-mini` (OpenAI) - Great value for most tasks
- `deepseek-chat` (DeepSeek) - Excellent quality-to-price ratio
- `gemini-2.0-flash` (Google) - Large context window
**Premium Options:**
- `gpt-4o` (OpenAI) - Excellent tool calling
- `claude-3.5-sonnet` (Anthropic) - High-quality reasoning
- `grok-3` (xAI) - Cutting-edge intelligence
#### Embedding Models (Search & Similarity)
**Recommended:**
- `text-embedding-3-small` (OpenAI) - $0.02 per 1M tokens
- `text-embedding-004` (Google) - Generous free tier
- `mistral-embed` (Mistral) - European alternative
#### Text-to-Speech (Podcast Generation)
**High Quality:**
- `eleven_turbo_v2_5` (ElevenLabs) - Best voice quality
- `tts-1` (OpenAI) - Good quality, reliable
**Budget Options:**
- `gemini-2.5-flash-preview-tts` (Google) - $10 per 1M tokens
#### Speech-to-Text (Audio Transcription)
**Recommended:**
- `whisper-1` (OpenAI) - Industry standard
- `scribe_v1` (ElevenLabs) - High-quality transcription
### 3. Provider-Specific Setup
#### OpenAI Setup
1. Visit https://platform.openai.com/
2. Create account and navigate to **API Keys**
3. Click **"Create new secret key"**
4. Add at least $5 in billing credits
5. Copy key to your `.env` file
#### Anthropic Setup
1. Visit https://console.anthropic.com/
2. Create account and navigate to **API Keys**
3. Generate new key
4. Add to environment variables
#### Google (Gemini) Setup
1. Visit https://makersuite.google.com/app/apikey
2. Create new API key
3. Add to environment variables
### 4. Model Recommendations by Use Case
#### Personal Research
```env
# Language: gpt-4o-mini (OpenAI)
# Embedding: text-embedding-3-small (OpenAI)
# TTS: tts-1 (OpenAI)
# STT: whisper-1 (OpenAI)
```
#### Professional Use
```env
# Language: claude-3.5-sonnet (Anthropic)
# Embedding: text-embedding-004 (Google)
# TTS: eleven_turbo_v2_5 (ElevenLabs)
# STT: whisper-1 (OpenAI)
```
#### Budget-Conscious
```env
# Language: deepseek-chat (DeepSeek)
# Embedding: text-embedding-004 (Google)
# TTS: gemini-2.5-flash-preview-tts (Google)
# STT: whisper-1 (OpenAI)
```
---
## Verification and Testing
### 1. Service Health Checks
#### Check All Services
```bash
# For source installation
make status
# For Docker
docker compose ps
```
#### Individual Service Tests
```bash
# Test database connection
curl http://localhost:8000/health
# Test API backend
curl http://localhost:5055/health
# Test Streamlit UI
curl http://localhost:8502/healthz
```
### 2. Create Test Notebook
1. **Access Web Interface**
- Open `http://localhost:8502`
- You should see the Open Notebook home page
2. **Create First Notebook**
- Click "Create New Notebook"
- Name: "Test Notebook"
- Description: "Testing installation"
- Click "Create"
3. **Add Test Source**
- Click "Add Source"
- Select "Text" tab
- Paste: "This is a test document for Open Notebook installation."
- Click "Add Source"
4. **Test Chat Function**
- Go to Chat tab
- Ask: "What is this document about?"
- You should receive a response about the test document
### 3. Feature Testing
#### Test Search Functionality
1. Add multiple sources to your notebook
2. Use the search bar to find specific content
3. Verify both full-text and semantic search work
#### Test Transformations
1. Select a source
2. Click "Transform" → "Summarize"
3. Verify transformation completes successfully
#### Test Podcast Generation
1. Add substantial content to your notebook
2. Navigate to "Podcast" tab
3. Click "Generate Podcast"
4. Wait for background processing to complete
---
## Security Configuration
### Password Protection
For public deployments, enable password protection:
```env
# Add to your .env or docker.env file
OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
```
**Features:**
- **Streamlit UI**: Password prompt on first access
- **REST API**: Requires `Authorization: Bearer your_password` header
- **Local Usage**: Optional (can be left empty)
### API Security
When using the REST API programmatically:
```bash
# Example API call with password
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
```
### Network Security
For production deployments:
1. **Use HTTPS**: Configure reverse proxy (nginx, Cloudflare)
2. **Firewall Rules**: Restrict access to necessary ports only
3. **VPN Access**: Consider VPN for private networks
4. **Regular Updates**: Keep Docker images updated
```bash
# Update Docker images
docker compose pull
docker compose up -d
```
---
## Troubleshooting
### Common Installation Issues
#### Port Already in Use
```bash
# Error: Port 8502 is already in use
# Solution: Find and stop conflicting process
lsof -i :8502
kill -9 <PID>
# Or use different port
uv run --env-file .env streamlit run app_home.py --server.port=8503
```
#### Permission Denied (Docker)
```bash
# Error: Permission denied accessing Docker
# Solution: Add user to docker group
sudo usermod -aG docker $USER
# Log out and log back in
```
#### Python/uv Installation Issues
```bash
# Error: uv command not found
# Solution: Install uv package manager
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc
# Error: Python version conflict
# Solution: Use uv's Python management
uv python install 3.11
uv python pin 3.11
```
#### libmagic Installation Issues
```bash
# macOS: Install via Homebrew
brew install libmagic
# Ubuntu/Debian: Install dev package
sudo apt install libmagic-dev
# CentOS/RHEL/Fedora: Install dev package
sudo dnf install file-devel
```
### API and Database Issues
#### Database Connection Failed
```bash
# Check if SurrealDB is running
docker compose ps surrealdb
# Check database logs
docker compose logs surrealdb
# Restart database
docker compose restart surrealdb
```
#### API Backend Not Responding
```bash
# Check API logs
docker compose logs api
# For source installation
# Check if API process is running
pgrep -f "run_api.py"
# Restart API
make api
```
#### Worker Not Processing Jobs
```bash
# Check worker status
pgrep -f "surreal-commands-worker"
# Restart worker
make worker-restart
# Check worker logs
docker compose logs worker
```
### AI Provider Issues
#### OpenAI API Key Errors
```bash
# Error: Invalid API key
# Solution: Verify key format and billing
# 1. Check key starts with "sk-"
# 2. Verify billing credits in OpenAI dashboard
# 3. Check API key permissions
```
#### Model Not Available
```bash
# Error: Model not found
# Solution: Check model availability
# 1. Verify model name in provider documentation
# 2. Check API key permissions
# 3. Try alternative model
```
#### Rate Limiting Issues
```bash
# Error: Rate limit exceeded
# Solution: Implement backoff strategy
# 1. Reduce concurrent requests
# 2. Upgrade provider plan
# 3. Use multiple providers
```
### Performance Issues
#### Slow Response Times
```bash
# Check system resources
top
docker stats
# Optimize database
# Consider increasing Docker memory limits
# Use faster storage (SSD)
```
#### Memory Issues
```bash
# Error: Out of memory
# Solution: Increase Docker memory
# 1. Docker Desktop → Settings → Resources
# 2. Increase memory limit to 4GB+
# 3. Consider model optimization
```
### Data and Storage Issues
#### Persistent Data Loss
```bash
# Ensure volumes are properly mounted
docker compose config
# Check volume permissions
ls -la ./notebook_data
ls -la ./surreal_data
# Fix permissions if needed
sudo chown -R $USER:$USER ./notebook_data
sudo chown -R $USER:$USER ./surreal_data
```
### Getting Help
#### Community Support
- **Discord**: https://discord.gg/37XJPXfz2w
- **GitHub Issues**: https://github.com/lfnovo/open-notebook/issues
- **Installation Assistant**: https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant
#### Bug Reports
When reporting issues, include:
1. Installation method (Docker/source)
2. Operating system and version
3. Error messages and logs
4. Steps to reproduce
5. Environment configuration (without API keys)
#### Log Collection
```bash
# Collect all logs
docker compose logs > open-notebook-logs.txt
# For source installation
make status > status.txt
```
---
## Next Steps
After successful installation:
1. **Read the User Guide**: Learn about features and workflows
2. **Check Model Providers**: Explore different AI providers for your needs
3. **Configure Transformations**: Set up custom content processing
4. **Explore API**: Use the REST API for integrations
5. **Join Community**: Connect with other users for tips and support
### Advanced Configuration
For advanced users:
- **Custom Prompts**: Customize AI behavior with Jinja templates
- **API Integration**: Build custom applications using the REST API
- **Multi-User Setup**: Configure for team usage
- **Backup Strategy**: Set up automated backups
### Performance Optimization
- **Model Selection**: Choose optimal models for your use case
- **Caching**: Configure appropriate cache settings
- **Resource Limits**: Tune Docker resource allocation
- **Monitoring**: Set up logging and monitoring
Welcome to Open Notebook! 🚀