* fix(proxy): keep internal SurrealDB websocket out of HTTP proxy (#1160)
websockets 15.0 auto-detects HTTP_PROXY/HTTPS_PROXY and tunnels even
ws:// connections through the proxy. The SurrealDB SDK connects over a
websocket, so with a proxy set the internal DB connection was routed
through the external proxy, which rejected the internal host with HTTP
403 and killed the worker/API on startup.
- Add ensure_internal_no_proxy() helper that merges host.docker.internal,
surrealdb, localhost, 127.0.0.1 into no_proxy/NO_PROXY (never clobbering
a user value) and call it at API, worker and DB-module startup.
- Add host.docker.internal and surrealdb to the .env.example and docs
NO_PROXY examples.
- Add unit tests for the injection helper.
* fix(proxy): preserve NO_PROXY wildcard and include custom SurrealDB host (#1160)
Address review findings on the no_proxy injection:
- NO_PROXY=* (bypass all hosts) is now treated as terminal: leave the
user's config untouched instead of narrowing the wildcard to a finite
list by appending the internal hosts.
- Parse the SurrealDB host from SURREAL_URL (falling back to
SURREAL_ADDRESS) and add it to the bypass list, so deployments with a
custom DB host/IP no longer route DB traffic through the proxy. Unset
or malformed values fall back to the four defaults gracefully.
- Drop the inaccurate getproxies() caching remark in the test.
Docling and local Crawl4AI are now opt-in, installed on first container startup
via OPEN_NOTEBOOK_ENABLE_DOCLING / OPEN_NOTEBOOK_ENABLE_CRAWL4AI, keeping the
default image lean (Crawl4AI un-bundled). Downloads cache on the /app/data
volume; installs are blocking with loud logs and degrade-don't-die on failure.
A GET /api/capabilities probe reports actual availability and the Settings UI
gates the Docling/Crawl4AI engines and OCR toggle (with env-var hints) until the
runtime is present. Docs folded from #1121 and rewritten to opt-in; ADR-007 added.
Closes#1122. Closes#1105.
Unsupported files used to enqueue a background job that failed and then
burned the full 15-attempt retry budget (~1h) before showing a generic
"Failed" with no actionable detail.
- Add a pre-flight `_assert_file_supported()` using content-core 2.x's
header-only `check_file_support()` (same routing as real extraction) in
the upload branch of `_build_content_state`, before any job is enqueued
and before the source record is created.
- Guard the source-retry endpoint the same way. Unexpected check errors
(e.g. file removed before a retry) fall through to normal extraction.
- Map `UnsupportedTypeException` to `415 Unsupported Media Type` via a
dedicated global handler (previously fell through to the base
OpenNotebookError handler's 500). The message names the detected type.
Part of #939.
Migration 22 best-effort maps profiles whose outline_llm/transcript_llm/
voice_model references are still empty to existing model records
(provider + name + type, no auto-create), clears the legacy values and
drops the 6 columns. The startup data migration that retried this
mapping on every boot (open_notebook/podcasts/migration.py) is deleted
along with its api/main.py lifespan hook; the legacy fields are removed
from the Pydantic models, API schemas and frontend types/panels.
Accepted trade-off: profiles whose mapping never converged lose the
legacy strings and stay unresolved - they were already non-functional
and the UI already flags them; the user re-picks models once.
Consolidate the three copies of context assembly into
open_notebook/utils/context_builder.py:
- POST /api/chat/context now delegates to build_notebook_context()
(same request/response shapes, same string-matching config semantics)
- The source-chat graph now calls build_source_context() instead of the
495-line generalized ContextBuilder class, which had exactly one
caller and whose notebook/notes/priority-config flexibility was dead
- POST /api/notebooks/{notebook_id}/context removed: it duplicated
/api/chat/context with a slightly different envelope and had zero
callers (frontend, docs, tests)
Behavior is pinned by new characterization tests written before the
refactor (tests/test_context_endpoint_characterization.py) plus unit
tests for build_source_context.
* refactor(ai): single provider registry as the backend source of truth
Provider metadata (env vars, modalities, connection-test models,
OpenAI-compatible discovery URLs, display names, docs links) is now
defined once in open_notebook/ai/provider_registry.py. The existing
surfaces are derived from it, keeping every import and call-site shape
unchanged:
- api/credentials_service.py: PROVIDER_ENV_CONFIG, PROVIDER_MODALITIES
and the discovery url_map are built from the registry
- open_notebook/ai/connection_tester.py: TEST_MODELS derived
- open_notebook/ai/model_discovery.py: OPENAI_COMPAT_PROVIDERS built
from registry entries with a discovery URL (quirk hooks stay local)
The SupportedProvider Literal (typing, can't be built at runtime) and
the frontend provider tables remain manual copies; the cross-check
tests now assert registry keys == Literal == frontend list, plus
registry internal consistency and discovery-table coverage.
New GET /api/providers endpoint exposes the registry (name, display
name, modalities, docs_url, env-configured status) so clients can stop
hardcoding provider lists (frontend adoption is a follow-up).
Docs updated: open_notebook/AGENTS.md and docs/7-DEVELOPMENT/credentials.md
now describe the registry instead of the four-place sync rule.
* refactor(ai): address review findings on the provider registry
- Build PROVIDERS via _build_registry(), which raises on a duplicate
provider name at import time instead of silently dropping the earlier
spec (dict-comprehension behavior); regression test added
- Pin the exact OpenAI-compatible provider -> discovery URL mapping in
a test so a registry edit can't silently drop or misassign a URL
- Give TEST_MODELS a real type annotation
(Dict[str, Tuple[Optional[str], str]]) instead of bare dict
* fix: reject oversized request bodies before auth/routing
No limit existed on request body size, so a single upload could exhaust
memory/disk before any validation ran. MaxBodySizeMiddleware
(api/middleware.py) rejects requests over OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB
(default 100MB) via both a Content-Length pre-check and by counting bytes
as the body streams in, so it also catches chunked requests with no
Content-Length header.
Registered after PasswordAuthMiddleware (wrapping it) so oversized
requests are rejected before spending any work on credential checks, and
inside CORSMiddleware so a rejected upload still gets CORS headers.
* fix: clamp non-positive size limits and log rejected requests
OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB=0 (or negative) made every request with a
body 413; fall back to the default instead, with a warning. Also log each
413 rejection so operators can diagnose failed uploads.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: don't combine wildcard CORS origins with allow_credentials
Combining allow_origins=["*"] with allow_credentials=True makes
Starlette's CORSMiddleware reflect the request's Origin header verbatim
instead of returning a literal "*" (browsers reject a literal wildcard
alongside credentials) - defeating the origin allowlist for any
credentialed request.
allow_credentials is now tied to whether CORS_ORIGINS was explicitly
scoped: False for the default wildcard, True once an operator opts into
specific origins. _cors_headers() (the manual CORS builder for error
responses raised before CORSMiddleware runs) is updated to match, so it
can't grant credentials the real middleware wouldn't.
Not independently exploitable today (the frontend never sends
credentialed requests, and auth is a Bearer header, not a cookie), but
there's no reason to allow it for the default wildcard case.
* fix: key allow_credentials on the parsed origins list, not the env var
An operator who explicitly sets CORS_ORIGINS=* got allow_credentials=True
with a wildcard origin list - the exact reflect-any-Origin behavior this
change exists to prevent. Introduce CORS_ALLOW_CREDENTIALS keyed on the
parsed list containing '*' and use it at both the middleware registration
and the manual error-response headers; replace the tautological formula
tests with ones exercising the real parser.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: make API startup resilient to a not-yet-ready database
Fixes#708
* fix: bound the database readiness probe with a per-attempt timeout
_wait_for_database awaited migration_manager.ping() with no timeout, so a
hung connection probe could bypass the retry budget and block startup
indefinitely. Wrap the probe in asyncio.wait_for with a per-attempt ceiling;
a timed-out probe is treated as a transient failure and retried like any
other unreachable-database attempt, keeping the retry budget bounded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for startup database readiness retry (#708)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Replace hardcoded `allow_origins=["*"]` with a parsed `CORS_ORIGINS`
environment variable (comma-separated). Default remains `*` for
backward compatibility — no existing deployment breaks — but the API
now logs a startup warning prompting users to set it explicitly for
production.
Exception handlers now route their CORS headers through a shared
`_cors_headers()` helper that mirrors Starlette's CORSMiddleware
behavior: reflects the request Origin when allowed (handling the
browser-rejected `*` + credentials combination correctly), and omits
`Access-Control-Allow-Origin` for disallowed origins so error bodies
don't leak cross-origin when `CORS_ORIGINS` is configured.
Closes#585, #730.
Based on the original work by Greg Grace in #597; rewritten on top of
current main to address prior review feedback (load_dotenv kept at
top, `import os` grouped with stdlib, `_cors_headers` defined before
its exception-handler callers, origins parsed once at module load)
and to choose a non-breaking default paired with a startup warning
instead of a stricter-by-default origin.
Co-authored-by: Greg Grace <ggrace@519lab.com>
* feat(podcasts): integrate model registry for profiles and credential passthrough
Replace loose provider/model string fields with record<model> references
in podcast profiles, enabling credential passthrough to podcast-creator.
Backend:
- EpisodeProfile: outline_llm, transcript_llm (record<model>) replace
outline_provider/outline_model strings. New language field (BCP 47).
- SpeakerProfile: voice_model (record<model>) replaces tts_provider/
tts_model strings. Per-speaker voice_model override support.
- Migration 14: schema changes making legacy fields optional, adding new
record<model> fields.
- Data migration (migration.py): auto-converts legacy profiles to model
registry references on startup. Idempotent.
- podcast_commands.py: resolves credentials for ALL profiles before
calling podcast-creator.
- New /api/languages endpoint (pycountry + babel) with BCP 47 locale
codes (pt-BR, en-US, etc.).
Frontend:
- Episode/speaker profile forms use ModelSelector instead of manual
provider/model dropdowns.
- Language dropdown with BCP 47 codes in episode profile form.
- Per-speaker TTS voice model override in speaker profile form.
- "Templates" tab renamed to "Profiles".
- Setup required badge on unconfigured profiles.
- i18n updated across all 8 locales.
Closes#486, closes#552
* fix(i18n): remove unused legacy podcast provider/model keys
Remove 10 orphaned i18n keys across all 8 locales that were left behind
after replacing manual provider/model dropdowns with ModelSelector.
* fix: address review violations in podcast model registry
- P1: Remove profiles with failed model resolution from dicts to prevent
podcast-creator validation errors on unrelated profiles
- P2: Use centralized QUERY_KEYS.languages instead of inline key
- P3: Fix ISO 639-1 → BCP 47 in model field description and CLAUDE.md
- P3: Update "templates" → "profiles" in locale string values (all 8)
* chore: bump version to 1.8.0
Replace generic "An unexpected error occurred" messages with descriptive,
user-friendly error messages when LLM operations fail. Errors like invalid
API keys, wrong model names, and rate limits now surface clearly in the UI.
Adds error classification utility, global FastAPI exception handlers, and
frontend getApiErrorMessage() helper. Bumps version to 1.7.2.
* feat: replace provider config with credential-based system (#477)
Introduce a new credential management system replacing the old
ProviderConfig singleton and standalone Models page. Each credential
stores encrypted API keys and provider-specific configuration with
full CRUD support via a unified settings UI.
Backend:
- Add Credential domain model with encrypted API key storage
- Add credentials API router (CRUD, discovery, registration, testing)
- Add encryption utilities for secure key storage
- Add key_provider for DB-first env-var fallback provisioning
- Add connection tester and model discovery services
- Integrate ModelManager with credential-based config
- Add provider name normalization for Esperanto compatibility
- Add database migrations 11-12 for credential schema
Frontend:
- Rewrite settings/api-keys page with credential management UI
- Add model discovery dialog with search and custom model support
- Add compact default model assignments (primary/advanced layout)
- Add inline model testing and credential connection testing
- Add env-var migration banner
- Update navigation to unified settings page
- Remove standalone models page and old settings components
i18n:
- Update all 7 locale files with credential and model management keys
Closes#477
Co-Authored-By: JFMD <git@jfmd.us>
Co-Authored-By: OraCatQAQ <570768706@qq.com>
* fix: address PR #540 review comments
- Fix docs referencing removed Models page
- Fix error-handler returning raw messages instead of i18n keys
- Fix auth.py misleading docstring and missing no-password guard
- Fix connection_tester using wrong env var for openai_compatible
- Add provision_provider_keys before model discovery/sync
- Update CLAUDE.md to reflect credential-based system
- Fix missing closing brace in api-keys page useEffect
* fix: add logging to credential migration and surface errors in UI
- Add comprehensive logging to migrate-from-env and
migrate-from-provider-config endpoints (start, per-provider
progress, success/failure with stack traces, final summary)
- Fix frontend migration hooks ignoring errors array from response
- Show error toast when migration fails instead of "nothing to migrate"
- Invalidate status/envStatus queries after migration so banner updates
* docs: update CLAUDE.md files for credential system
Replace stale ProviderConfig and /api-keys/ references across 8 CLAUDE.md
files to reflect the new Credential-based system from PR #540.
* docs: update user documentation for credential-based system
Replace env var API key instructions with Settings UI credential
workflow across all user-facing documentation. The new flow is:
set OPEN_NOTEBOOK_ENCRYPTION_KEY → start services → add credential
in Settings UI → test → discover models → register.
- Rewrite ai-providers.md, api-configuration.md, environment-reference.md
- Update all quick-start guides and installation docs
- Update ollama.md, openai-compatible.md, local-tts/stt networking sections
- Update reverse-proxy.md, development-setup.md, security.md
- Fix broken links to non-existent docs/deployment/ paths
- Add credentials endpoints to api-reference.md
- Move all API key env vars to deprecated/legacy sections
* chore: bump version to 1.7.0-rc1
Release candidate for credential-based provider management system.
* fix: initialize provider before try block in test_credential
Prevents UnboundLocalError when Credential.get() throws (e.g.,
invalid credential_id) before provider is assigned.
* fix: reorder down migration to drop index before table
Removes duplicate REMOVE FIELD statement and reorders so the index
is dropped before the table, preventing rollback failures.
* refactor: simplify encryption key to always derive via SHA-256
Remove the dual code path in _ensure_fernet_key() that detected native
Fernet keys. Since the credential system is new, always deriving via
SHA-256 removes unnecessary complexity. Also removes the generate_key()
function and Fernet.generate_key() references from docs.
* fix: correct mock patch targets in embedding tests and URL validation
Fix embedding tests patching wrong module path for model_manager
(was targeting open_notebook.utils.embedding.model_manager but it's
imported locally from open_notebook.ai.models). Also fix URL validation
to allow unresolvable hostnames since they may be valid in the
deployment environment (e.g., Azure endpoints, internal DNS).
* feat: add global setup banner for encryption and migration status
Show a persistent banner in AppShell when encryption key is missing
(red) or env var API keys can be migrated (amber), so users see
these prompts on every page instead of only on Settings > API Keys.
Includes a docs link for the encryption banner and i18n support
across all 7 locales.
* docs: several improvements to docker-compose e env examples
* Update README.md
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
* docs: fix env var format in README and update model setup instructions
Align the encryption key snippet in README Step 2 with the list
format used in the compose file. Replace deprecated "Settings →
Models" instructions with credential-based Discover Models flow.
* fix: address credential system review issues
- Fix SSRF bypass via IPv4-mapped IPv6 addresses (::ffff:169.254.x.x)
- Fix TTS connection test missing config parameter
- Add Azure-specific model discovery using api-key auth header
- Add Vertex static model list for credential-based discovery
- Fix PROVIDER_DISCOVERY_FUNCTIONS incorrect azure/vertex mapping
- Extract business logic to api/credentials_service.py (service layer)
- Move credential Pydantic schemas to api/models.py
- Update tests to use new service imports and ValueError assertions
* fix: sanitize error responses and migrate key_provider to Credential
- Replace raw exception messages in all credential router 500 responses
with generic error strings (internal details logged server-side only)
- Refactor key_provider.py to use Credential.get_by_provider() instead
of deprecated ProviderConfig.get_instance()
- Remove unused functions (get_provider_configs, get_default_api_key,
get_provider_config) that were dead code
---------
Co-authored-by: JFMD <git@jfmd.us>
Co-authored-by: OraCatQAQ <570768706@qq.com>
* feat: decrease chunking size for maximum ollama compatibility
* docs: improve i18n info on Claude.md
* feat: add cascade deletion for notebooks with delete preview
- Add Notebook.get_delete_preview() to show counts of affected items
- Add Notebook.delete(delete_exclusive_sources) for cascade deletion
- Always delete notes when notebook is deleted
- Allow user to choose: delete or keep exclusive sources
- Shared sources are always unlinked but never deleted
- Add NotebookDeleteDialog component with radio button options
- Add delete-preview API endpoint
- Update delete endpoint with delete_exclusive_sources param
- Add i18n support for all 5 locales
Closes#77
* docs: remove harcoded config settings
- Added custom exception handler to ensure CORS headers are included in
all HTTP error responses from the API
- Added documentation for 413 (Payload Too Large) errors when behind
reverse proxies (nginx, traefik, kubernetes ingress)
- Added client_max_body_size to nginx configuration examples
- Documented how to configure CORS headers for proxy-level error responses
Fixes#401
New front-end
Launch Chat API
Manage Sources
Enable re-embedding of all contents
Sources can be added without a notebook now
Improved settings
Enable model selector on all chats
Background processing for better experience
Dark mode
Improved Notes
Improved Docs:
- Remove all Streamlit references from documentation
- Update deployment guides with React frontend setup
- Fix Docker environment variables format (SURREAL_URL, SURREAL_PASSWORD)
- Update docker image tag from :latest to :v1-latest
- Change navigation references (Settings → Models to just Models)
- Update development setup to include frontend npm commands
- Add MIGRATION.md guide for users upgrading from Streamlit
- Update quick-start guide with correct environment variables
- Add port 5055 documentation for API access
- Update project structure to reflect frontend/ directory
- Remove outdated source-chat documentation files
Creates the API layer for Open Notebook
Creates a services API gateway for the Streamlit front-end
Migrates the SurrealDB SDK to the official one
Change all database calls to async
New podcast framework supporting multiple speaker configurations
Implement the surreal-commands library for async processing
Improve docker image and docker-compose configurations
