open-notebook/open_notebook/CLAUDE.md

Open Notebook Core Backend

The open_notebook module is the heart of the system: a multi-layer backend orchestrating AI-powered research workflows. It bridges domain models, asynchronous database operations, LangGraph-based content processing, and multi-provider AI model management.

Purpose

Encapsulates the entire backend architecture:

1. Data layer: SurrealDB persistence with async CRUD and migrations
2. Domain layer: Research models (Notebook, Source, Note, etc.) with embedded relationships
3. Workflow layer: LangGraph state machines for content ingestion, chat, and transformations
4. AI provisioning: Multi-provider model management with smart fallback logic
5. Support services: Context building, tokenization, and utility functions

All components communicate through async/await patterns and use Pydantic for validation.

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    API / Streamlit UI                        │
└──────────────────────┬──────────────────────────────────────┘
                       │
    ┌──────────────────┴──────────────────┐
    │                                     │
┌───▼────────────────────┐   ┌──────────▼────────────────┐
│    Graphs (LangGraph)   │   │   Domain Models (Data)    │
│ - source.py (ingestion) │   │ - Notebook, Source, Note  │
│ - chat.py              │   │ - ChatSession, Asset       │
│ - ask.py (search)      │   │ - SourceInsight, Embedding│
│ - transformation.py    │   │ - Transformation, Settings│
└───┬────────────────────┘   │ - EpisodeProfile, Podcast │
    │                        └──────────┬─────────────────┘
    │                                   │
    └───────────────────┬───────────────┘
                        │
    ┌───────────────────┴────────────────────┐
    │                                        │
┌───▼─────────────────┐      ┌──────────────▼──────┐
│  AI Module (Models)  │      │  Utils (Helpers)     │
│ - ModelManager       │      │ - ContextBuilder     │
│ - DefaultModels      │      │ - TokenUtils         │
│ - provision_langchain│      │ - TextUtils          │
│ - Multi-provider AI  │      │ - VersionUtils       │
└───┬─────────────────┘      └──────────┬──────────┘
    │                                   │
    └───────────────────┬───────────────┘
                        │
         ┌──────────────▼────────────────┐
         │  Database (SurrealDB)          │
         │ - repository.py (CRUD ops)     │
         │ - async_migrate.py (schema)    │
         │ - Configuration                │
         └────────────────────────────────┘

Component Catalog

Core Layers

See dedicated CLAUDE.md files for detailed patterns and usage:

• database/: Async repository pattern (repo_query, repo_create, repo_upsert), connection pooling, and automatic schema migrations on API startup. See database/CLAUDE.md.

• domain/: Core data models using Pydantic with SurrealDB persistence. Two base classes: ObjectModel (mutable records with auto-increment IDs and embedding) and RecordModel (singleton configuration). Includes search functions (text_search, vector_search). See domain/CLAUDE.md.

• graphs/: LangGraph state machines for async workflows. Content ingestion (source.py), conversational agents (chat.py), search synthesis (ask.py), and transformations. Uses provision_langchain_model() for smart model selection with token-aware fallback. See graphs/CLAUDE.md.

• ai/: Centralized AI model lifecycle via Esperanto library. ModelManager factory with intelligent fallback (large context detection, type-specific defaults, config override). Supports 8+ providers (OpenAI, Anthropic, Google, Groq, Ollama, Mistral, DeepSeek, xAI). See ai/CLAUDE.md.

• utils/: Cross-cutting utilities: ContextBuilder (flexible context assembly from sources/notes/insights with token budgeting), TextUtils (truncation, cleaning), TokenUtils (GPT token counting), VersionUtils (schema compatibility). See utils/CLAUDE.md.

• podcasts/: Podcast generation models: SpeakerProfile (TTS voice config), EpisodeProfile (generation settings), PodcastEpisode (job tracking via surreal-commands). See podcasts/CLAUDE.md.

Configuration & Exceptions

• config.py: Paths for data folder, uploads, LangGraph checkpoints, and tiktoken cache. Auto-creates directories.
• exceptions.py: Hierarchy of OpenNotebookError subclasses for database, file, network, authentication, and rate-limit failures.

Data Flow: Content Ingestion

User uploads file/URL
         │
         ▼
┌─────────────────────────────────────┐
│ source.py (LangGraph state machine) │
├─────────────────────────────────────┤
│ 1. content_process()                │
│    - extract_content() from file/URL│
│    - Use ContentSettings defaults    │
│    - speech_to_text model from DB   │
│                                     │
│ 2. save_source()                    │
│    - Update Source with full_text   │
│    - Preserve title if empty        │
│                                     │
│ 3. trigger_transformations()        │
│    - Parallel fan-out to each TXN   │
└────────────────┬────────────────────┘
                 │
                 ▼
         ┌──────────────┐
         │ transformation.py (parallel)
         │ - Apply prompt to source text
         │ - Generate insights
         │ - Auto-embed results
         └──────────────┘
                 │
                 ▼
        ┌────────────────────┐
        │ Database Storage    │
        │ - Source.full_text  │
        │ - SourceInsight     │
        │ - Embeddings        │
        │ - (async job)       │
        └────────────────────┘

Fire-and-forget embeddings: Source.vectorize() returns command_id without awaiting; embedding happens asynchronously via surreal-commands job system.

Data Flow: Chat & Search

User message in chat
         │
         ▼
┌──────────────────────────┐
│ ContextBuilder           │
│ - Select sources/notes   │
│ - Token budget limiting  │
│ - Priority weighting     │
└──────────┬───────────────┘
           │
           ▼
┌──────────────────────────────────┐
│ chat.py or ask.py (LangGraph)    │
│ - Load context from above        │
│ - provision_langchain_model()    │
│   * Auto-upgrade for large text  │
│   * Apply model_id override      │
│ - Call LLM with context          │
│ - Store message in SqliteSaver   │
└──────────┬───────────────────────┘
           │
           ▼
    ┌──────────────┐
    │ LLM Response │
    │ (persisted)  │
    └──────────────┘

Key Patterns Across Layers

Async/Await Everywhere

All database operations, model provisioning, and graph execution are async. Mix with sync code only via asyncio.run() or LangGraph's async bridges (see graphs/CLAUDE.md for workarounds).

Type-Driven Dispatch

Model types (language, embedding, speech_to_text, text_to_speech) drive factory logic in ModelManager. Domain model IDs encode their type: notebook:uuid, source:uuid, note:uuid.

Smart Fallback Logic

provision_langchain_model() auto-detects large contexts (105K+ tokens) and upgrades to dedicated large_context_model. Falls back to default_chat_model if specific type not found.

Fire-and-Forget Jobs

Time-consuming operations (embedding, podcast generation) return command_id immediately. Caller polls surreal-commands for status; no blocking.

Fire-and-Forget Embedding

Domain models submit embedding commands after save via submit_command() (non-blocking). Note.save() submits embed_note, Source.add_insight() submits embed_insight, Source.vectorize() submits embed_source. Search functions (text_search, vector_search) use embeddings for semantic matching.

Relationship Management

SurrealDB graph edges link entities: Notebook→Source (has), Source→Note (artifact), Note→Source (refers_to). See relate() in domain/base.py.

Integration Points

API startup (api/main.py):

• AsyncMigrationManager.run_migration_up() on lifespan startup
• Ensures schema is current before handling requests

Streamlit UI (pages/stream_app/):

• Calls domain models directly to fetch/create notebooks, sources, notes
• Invokes graphs (chat, source, ask) via async wrapper
• Relies on API for migrations (deprecated check in UI)

Background Jobs (surreal_commands):

• Source.vectorize() submits async embedding job
• PodcastEpisode.get_job_status() polls job queue
• Decouples long-running operations from request flow

Important Quirks & Gotchas

1. Token counting rough estimate: Uses cl100k_base encoding; may differ 5-10% from actual model
2. Large context threshold hard-coded: 105,000 token limit for large_context_model upgrade (not configurable)
3. Async loop gymnastics in graphs: ThreadPoolExecutor workaround for LangGraph sync nodes calling async functions (fragile)
4. DefaultModels always fresh: get_instance() bypasses singleton cache to pick up live config changes
5. Polymorphic model.get(): Resolves subclass from ID prefix; fails silently if subclass not imported
6. RecordID string inconsistency: repo_update() accepts both "table:id" format and full RecordID
7. Snapshot profiles: podcast profiles stored as dicts, so config updates don't affect past episodes
8. No connection pooling: Each repo_* creates new connection (adequate for HTTP but inefficient for bulk)
9. Circular import guard: utils imports domain; domain must not import utils (breaks on import)
10. SqliteSaver shared location: LangGraph checkpoints from LANGGRAPH_CHECKPOINT_FILE env var; all graphs use same file

How to Add New Feature

New data model:

1. Create class inheriting from ObjectModel with table_name ClassVar
2. Define Pydantic fields and validators
3. Override save() to submit embedding command if searchable (use submit_command("embed_*", id))
4. Add custom methods for domain logic (get_X, add_to_Y)
5. Register in domain/init.py exports

New workflow:

1. Create state machine in graphs/WORKFLOW.py using StateGraph
2. Import domain models and provision_langchain_model()
3. Define nodes as async functions taking State, returning dict
4. Compile with graph.compile()
5. Invoke from API endpoint or Streamlit page

New AI model type:

1. Add type string to Model class
2. Add AIFactory.create_* method in Esperanto
3. Handle in ModelManager.get_model()
4. Add DefaultModels field + getter

Key Dependencies

• surrealdb: AsyncSurreal client, RecordID type
• pydantic: Validation, field_validator
• langgraph: StateGraph, Send, SqliteSaver, async/sync bridging
• langchain_core: Messages, OutputParser, RunnableConfig
• esperanto: Multi-provider AI model abstraction (OpenAI, Anthropic, Google, Groq, Ollama, etc.)
• content-core: File/URL content extraction
• ai_prompter: Jinja2 template rendering for prompts
• surreal_commands: Async job queue for embeddings, podcast generation
• loguru: Structured logging throughout
• tiktoken: GPT token encoding for context window estimation

Codebase Statistics

• Modules: 6 core layers + support services
• Async operations: Database, AI provisioning, graph execution, embedding, job tracking
• Supported AI providers: 8+ (OpenAI, Anthropic, Google, Groq, Ollama, Mistral, DeepSeek, xAI, OpenRouter)
• Domain models: Notebook, Source, Note, SourceInsight, SourceEmbedding, ChatSession, Asset, Transformation, ContentSettings, EpisodeProfile, SpeakerProfile, PodcastEpisode
• Graph workflows: 6 (source, chat, source_chat, ask, transformation, prompt)