open-notebook/api/CLAUDE.md

# API Module

FastAPI-based REST backend exposing services for notebooks, sources, notes, chat, podcasts, and AI model management.

## Purpose

FastAPI application serving three architectural layers: routes (HTTP endpoints), services (business logic), and models (request/response schemas). Integrates LangGraph workflows (chat, ask, source_chat), SurrealDB persistence, and AI providers via Esperanto.

## Architecture Overview

**Three layers**:
1. **Routes** (`routers/*`): HTTP endpoints mapping to services
2. **Services** (`*_service.py`): Business logic orchestrating domain models, database, graphs, AI providers
3. **Models** (`models.py`): Pydantic request/response schemas with validation

**Startup flow**:
- Load .env environment variables
- Initialize CORS middleware + password auth middleware
- Run database migrations via AsyncMigrationManager on lifespan startup
- Register all routers

**Key services**:
- `chat_service.py`: Invokes chat graph with messages, context
- `podcast_service.py`: Orchestrates outline + transcript generation
- `sources_service.py`: Content ingestion, vectorization, metadata
- `notes_service.py`: Note creation, linking to sources/insights
- `transformations_service.py`: Applies transformations to content
- `models_service.py`: Manages AI provider/model configuration
- `episode_profiles_service.py`: Manages podcast speaker/episode profiles

## Component Catalog

### Main Application
- **main.py**: FastAPI app initialization, CORS setup, auth middleware, lifespan event, router registration
- **Lifespan handler**: Runs AsyncMigrationManager on startup (database schema migration)
- **Auth middleware**: PasswordAuthMiddleware protects endpoints (password-based access control)

### Services (Business Logic)
- **chat_service.py**: Invokes chat.py graph; handles message history via SqliteSaver
- **podcast_service.py**: Generates outline (outline.jinja), then transcript (transcript.jinja) for episodes
- **sources_service.py**: Ingests files/URLs (content_core), extracts text, vectorizes, saves to SurrealDB
- **transformations_service.py**: Applies transformations via transformation.py graph
- **models_service.py**: Manages ModelManager config (AI provider overrides)
- **episode_profiles_service.py**: CRUD for EpisodeProfile and SpeakerProfile models
- **insights_service.py**: Generates and retrieves source insights
- **notes_service.py**: Creates notes linked to sources/insights

### Models (Schemas)
- **models.py**: Pydantic schemas for request/response validation
- Request bodies: ChatRequest, CreateNoteRequest, PodcastGenerationRequest, etc.
- Response bodies: ChatResponse, NoteResponse, PodcastResponse, etc.
- Custom validators for enum fields, file paths, model references

### Routers
- **routers/chat.py**: POST /chat
- **routers/source_chat.py**: POST /source/{source_id}/chat
- **routers/podcasts.py**: POST /podcasts, GET /podcasts/{id}, etc.
- **routers/notes.py**: POST /notes, GET /notes/{id}
- **routers/sources.py**: POST /sources, GET /sources/{id}, DELETE /sources/{id}
- **routers/models.py**: GET /models, POST /models/config
- **routers/credentials.py**: CRUD + test + discover + migrate for credential management
- **routers/transformations.py**: POST /transformations
- **routers/insights.py**: GET /sources/{source_id}/insights
- **routers/auth.py**: POST /auth/password (password-based auth)
- **routers/commands.py**: GET /commands/{command_id} (job status tracking)

## Common Patterns

- **Service injection via FastAPI**: Routers import services directly; no DI framework
- **Async/await throughout**: All DB queries, graph invocations, AI calls are async
- **SurrealDB transactions**: Services use repo_query, repo_create, repo_upsert from database layer
- **Config override pattern**: Models/config override via models_service passed to graph.ainvoke(config=...)
- **Error handling**: Services catch exceptions and return HTTP status codes (400 Bad Request, 404 Not Found, 500 Internal Server Error)
- **Logging**: loguru logger in main.py; services expected to log key operations
- **Response normalization**: All responses follow standard schema (data + metadata structure)

## Key Dependencies

- `fastapi`: FastAPI app, routers, HTTPException
- `pydantic`: Validation models with Field, field_validator
- `open_notebook.graphs`: chat, ask, source_chat, source, transformation graphs
- `open_notebook.database`: SurrealDB repository functions (repo_query, repo_create, repo_upsert)
- `open_notebook.domain`: Notebook, Source, Note, SourceInsight models
- `open_notebook.ai.provision`: provision_langchain_model() factory
- `ai_prompter`: Prompter for template rendering
- `content_core`: extract_content() for file/URL processing
- `esperanto`: AI provider client library (LLM, embeddings, TTS)
- `surreal_commands`: Job queue for async operations (podcast generation)
- `loguru`: Structured logging

## Important Quirks & Gotchas

- **Migration auto-run**: Database schema migrations run on every API startup (via lifespan); no manual migration steps
- **PasswordAuthMiddleware is basic**: Uses simple password check; production deployments should replace with OAuth/JWT
- **No request rate limiting**: No built-in rate limiting; deployment must add via proxy/middleware
- **Service state is stateless**: Services don't cache results; each request re-queries database/AI models
- **Graph invocation is blocking**: chat/podcast workflows may take minutes; no timeout handling in services
- **Command job fire-and-forget**: podcast_service.py submits jobs but doesn't wait (async job queue pattern)
- **Model override scoping**: Model config override via RunnableConfig is per-request only (not persistent)
- **CORS open by default**: main.py CORS settings allow all origins (restrict before production)
- **No OpenAPI security scheme**: API docs available without auth (disable before production)
- **Services don't validate user permission**: All endpoints trust authentication layer; no per-notebook permission checks

## How to Add New Endpoint

1. Create router file in `routers/` (e.g., `routers/new_feature.py`)
2. Import router into `main.py` and register: `app.include_router(new_feature.router, tags=["new_feature"])`
3. Create service in `new_feature_service.py` with business logic
4. Define request/response schemas in `models.py` (or create `new_feature_models.py`)
5. Implement router functions calling service methods
6. Test with `uv run uvicorn api.main:app --host 0.0.0.0 --port 5055`

## Testing Patterns

- **Interactive docs**: http://localhost:5055/docs (Swagger UI)
- **Direct service tests**: Import service, call methods directly with test data
- **Mock graphs**: Replace graph.ainvoke() with mock for testing service logic
- **Database: Use test database** (separate SurrealDB instance or mock repo_query)

---

## Credential Management (API Configuration UI)

The Credential Management system enables users to configure AI provider credentials through the UI instead of environment variables. Keys are stored securely in SurrealDB (encrypted via Fernet) with database-first fallback to environment variables.

### Router: `routers/credentials.py`

**Endpoints**:

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/credentials` | List all credentials (optional `?provider=` filter) |
| GET | `/credentials/by-provider/{provider}` | List credentials for a provider |
| POST | `/credentials` | Create a new credential |
| GET | `/credentials/{credential_id}` | Get a specific credential |
| PUT | `/credentials/{credential_id}` | Update a credential |
| DELETE | `/credentials/{credential_id}` | Delete a credential |
| POST | `/credentials/{credential_id}/test` | Test connection using credential |
| POST | `/credentials/{credential_id}/discover` | Discover available models |
| POST | `/credentials/{credential_id}/register-models` | Register discovered models |
| POST | `/credentials/migrate-from-provider-config` | Migrate from legacy ProviderConfig |

**Supported Providers** (13 total):
- Simple API key: `openai`, `anthropic`, `google`, `groq`, `mistral`, `deepseek`, `xai`, `openrouter`, `voyage`, `elevenlabs`
- URL-based: `ollama`
- Multi-field: `azure`, `vertex`, `openai_compatible`

**Security Features**:
- NEVER returns actual API key values (only metadata)
- URL validation (SSRF protection) on all URL fields via `_validate_url()`
- Allows private IPs and localhost for self-hosted services (Ollama, LM Studio)
- Requires `OPEN_NOTEBOOK_ENCRYPTION_KEY` to be set for storing credentials

### Domain Model: `Credential` (`open_notebook/domain/credential.py`)

Individual credential records replacing the old `ProviderConfig` singleton. Each credential stores:
- Provider name, display name, modalities
- Encrypted API key (via Fernet)
- Provider-specific config (base_url, endpoint, api_version, etc.)

### Integration with Key Provider (`open_notebook/ai/key_provider.py`)

The `key_provider` module provisions DB-stored credentials into environment variables for Esperanto compatibility:

**Database-first Pattern**:
1. API endpoint saves keys to `Credential` records (encrypted in SurrealDB)
2. Before model provisioning, `provision_provider_keys(provider)` checks DB, then env vars
3. Keys from DB are set as environment variables for Esperanto compatibility
4. Existing env vars remain unchanged if no DB config exists

**Key Functions**:
- `get_api_key(provider)`: Get API key (DB first, env fallback)
- `provision_provider_keys(provider)`: Set env vars from DB for a provider
- `provision_all_keys()`: Load all provider keys from DB into env vars

### Authentication

No changes to authentication. The `credentials` router uses the same `PasswordAuthMiddleware` as all other endpoints. Keys are protected by the same password-based auth.

**Auth Flow** (unchanged from `api/auth.py`):
- `PasswordAuthMiddleware`: Global middleware checking `Authorization: Bearer {password}` header
- Default password: `open-notebook-change-me` (set `OPEN_NOTEBOOK_PASSWORD` in production)
- Docker secrets support via `OPEN_NOTEBOOK_PASSWORD_FILE`

### Connection Testing (`open_notebook/ai/connection_tester.py`)

The `/credentials/{credential_id}/test` endpoint uses minimal API calls to verify credentials:
- Loads Credential via `Credential.get(config_id)`, uses `credential.to_esperanto_config()`
- Uses cheapest/smallest models per provider (TEST_MODELS map)
- Returns success status and descriptive message
- Special handlers for ollama, openai_compatible, and azure providers

### Migration Workflows

Two migration endpoints help users transition to the credential system:

**From environment variables** (`POST /credentials/migrate-from-env`):
1. Checks each provider for env var presence
2. Creates Credential records from env var values
3. Returns summary: migrated, skipped, errors

**From legacy ProviderConfig** (`POST /credentials/migrate-from-provider-config`):
1. Reads old ProviderConfig records from database
2. Converts each to individual Credential records
3. Returns summary: migrated, skipped, errors

### Example Usage

```python
# Check status
GET /credentials/status
# Response: {"configured": {"openai": true, "anthropic": false}, "source": {"openai": "database", "anthropic": "none"}, "encryption_configured": true}

# Create credential
POST /credentials
{"name": "My OpenAI Key", "provider": "openai", "modalities": ["language", "embedding"], "api_key": "sk-proj-..."}

# Test connection
POST /credentials/{credential_id}/test
# Response: {"provider": "openai", "success": true, "message": "Connection successful"}

# Discover models
POST /credentials/{credential_id}/discover
# Response: {"provider": "openai", "models": [{"model_id": "gpt-4", "name": "gpt-4", ...}], "credential_id": "..."}

# Migrate from env
POST /credentials/migrate-from-env
# Response: {"message": "Migration complete. Migrated 3 providers.", "migrated": ["openai", "anthropic", "groq"], "skipped": [], "errors": []}
```