Surface content-core 2.x's docling_formulas (formula extraction) and
docling_vision (image/chart vision) enrichment flags in Settings →
Content Processing, mirroring the existing OCR toggle. Both default off
and are gated on Docling availability in the UI. The settings persist
via GET/PUT /api/settings and are threaded into content-core extraction
alongside docling_ocr. Migration 23 backfills the new fields on the
existing content_settings record. Labels and help are translated across
all 14 locales.
Closes#1131
Migration 22 best-effort maps profiles whose outline_llm/transcript_llm/
voice_model references are still empty to existing model records
(provider + name + type, no auto-create), clears the legacy values and
drops the 6 columns. The startup data migration that retried this
mapping on every boot (open_notebook/podcasts/migration.py) is deleted
along with its api/main.py lifespan hook; the legacy fields are removed
from the Pydantic models, API schemas and frontend types/panels.
Accepted trade-off: profiles whose mapping never converged lose the
legacy strings and stay unresolved - they were already non-functional
and the UI already flags them; the user re-picks models once.
* refactor(podcasts): store audio paths relative to PODCASTS_FOLDER
audio_file previously stored the absolute path returned by
podcast-creator (sometimes as a file:// URI), which kept path traversal
representable in the DB and broke playback when DATA_FOLDER moved.
- New single choke point open_notebook/podcasts/audio_paths.py:
to_relative_audio_path() validates at write time (the DB can never
hold an absolute or root-escaping value) and
resolve_contained_audio_path() joins + resolves + contains at read
time, replacing the per-router guards from #1018. Absolute/file://
legacy values are treated as invalid (same 403/404 as before).
- Generation command stores the relative form; in-band "ERROR: ..."
values from podcast-creator now fail the job with the real error.
- build_episode_output_dir() builds from PODCASTS_FOLDER so the write
root and the validation root cannot drift.
- Migration 21 converts legacy rows under the known roots (plain
file:/// URIs, /app/data/podcasts/, /data/podcasts/,
./data/podcasts/, data/podcasts/) via a single-strip IF/ELSE chain;
rows under other roots stay untouched by design.
Closes#1030
* test: use the tolerant migration-count assertion pattern
Per review: assert >= 21 with up/down parity and a distinctive-SQL check on
index 20, matching the convention from the migration-19/20 tests, so the next
migration doesn't trip an unrelated test file.
* fix(podcasts): reference speaker profiles by record ID instead of name
episode_profile.speaker_config now stores a record<speaker_profile>
reference (migration 20 converts existing rows; orphaned names become
null). The generate API keeps accepting the speaker profile by name and
resolves it to a record ID at the boundary.
Closes#630
* fix(podcasts): survive orphaned speaker references in podcast-creator config
Review fixes: rewrite record IDs back to speaker names (and drop orphaned
profiles) when building podcast-creator's episode config so one orphaned
profile can't fail validation for every generation; clearer error for
dangling references; resolve a single speaker name on single-record
endpoints instead of fetching the whole speaker table; accurate wording
about record links not preventing dangling references.
* fix(podcasts): submit the speaker profile record ID from the generate dialog
Submitting the cached display name made generation fail if the speaker
profile was renamed while the dialog was open - the record ID is stable
and the API resolves either form.
source_insight was the only content table without created/updated field
definitions. Since the table is SCHEMAFULL and insights are created via a
raw CREATE (create_insight command), SurrealDB silently dropped any
timestamps - rows genuinely held NONE - and the API then wrapped them in
str(), so clients received the literal string "None".
- Migration 19 defines created/updated on source_insight with the same
time::now() defaults used by source, note and notebook, so new insights
are stamped at creation. Existing rows are left untouched (no backfill).
- SourceInsightResponse.created/updated are now Optional[str]; both
routers emit an ISO 8601 string when the timestamp is present and null
when it is absent (legacy rows), never "None".
- Frontend types updated to string | null accordingly.
- New tests cover the migration definition/registration and the API
serialization (absent -> null, present -> ISO string).
Note: Python-side stamping alone was not viable - the table is SCHEMAFULL,
so undefined fields written by the client are silently dropped on
SurrealDB 2.x (verified against a live instance), hence the schema
migration mirroring the other tables.
Closes#1045
* fix: make API startup resilient to a not-yet-ready database
Fixes#708
* fix: bound the database readiness probe with a per-attempt timeout
_wait_for_database awaited migration_manager.ping() with no timeout, so a
hung connection probe could bypass the retry budget and block startup
indefinitely. Wrap the probe in asyncio.wait_for with a per-attempt ceiling;
a timed-out probe is treated as a transient failure and retried like any
other unreachable-database attempt, keeping the retry budget bounded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for startup database readiness retry (#708)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* feat: Recently Viewed section for notebooks and sources
Fixes#850
* fix: export RecentlyViewedResponse type from api types
The RecentlyViewed component and notebooks API client import
RecentlyViewedResponse from @/lib/types/api but the interface was missing,
failing the type-check build. Add it to match the backend api/models.py model.
* fix: harden recently-viewed read path and index the recency query
Three robustness fixes on the recently-viewed feature:
- Make the last_viewed_at write-on-read stamping best-effort. _stamp_source_view
and _stamp_notebook_view now swallow and log their own errors, so a failed
stamp update can no longer turn a successful GET /sources/{id} or
GET /notebooks/{id} into a 500.
- Stop leaking internal details from GET /recently-viewed: log the full
exception server-side and return a generic error message to the client.
- Add indexes on last_viewed_at for the notebook and source tables in
migration 16 so the ORDER BY last_viewed_at DESC LIMIT recently-viewed
query does not degrade into a full table scan as data grows. The down
migration removes the indexes before dropping the field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: register migration 18, add recently-viewed i18n keys and CHANGELOG entry
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* feat: per-transformation custom model selection
Fixes#776
* fix: validate model_id exists when creating or updating a transformation
The create and update endpoints persisted model_id without checking the
referenced model exists, so an invalid reference was stored silently and only
surfaced later as a 404 at execution time. Add the same existence check
execute_transformation already performs (Model.get -> 404 if missing) to both
the create and update paths, so a bad model_id is rejected up front with a
clear 404. Update still allows clearing model_id to None.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: cover model_id existence validation on transformation create/update
The create test now mocks Model.get so the new existence check passes and
asserts it was awaited with the supplied model_id. The update test expects
Model.get to be awaited twice (once validating the update, once at execute)
rather than once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for per-transformation model selection (#776)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: pass max_tokens through to podcast_creator for outline/transcript generation
* fix: persist and expose episode_profile max_tokens (migration + API)
* docs: add CHANGELOG entry for episode profile max_tokens pass-through (#639)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix(credentials): persist Ollama num_ctx via a flexible config object
The `credential` table is SCHEMAFULL, so the model's `num_ctx` field was
silently dropped on write and the override never took effect (#875).
Rather than add a typed column per provider option, add a single flexible
`config` object to the credential table (migration 15). Provider-specific
tuning options (currently `num_ctx`) are still exposed as top-level fields on
the Credential model and on the API, but are packed into `config` on save and
lifted back out on load via a before-validator. Future options only need a
Pydantic field + an entry in CONFIG_EXTRAS — no further migrations.
- migration 15 (+ down): DEFINE FIELD config ON credential FLEXIBLE TYPE option<object>
- Credential: CONFIG_EXTRAS set, _lift_config before-validator, config packing
in _prepare_save_data
- tests covering pack/unpack round-trip and the empty-config case
* fix(credentials): preserve unmapped config keys on save
Address cubic review on #903:
- `config` is now a real model field mirroring the credential table's FLEXIBLE
object and is the on-disk source of truth. `num_ctx` remains a convenience
field mirrored from/to config. On save we start from the existing bag and
sync the convenience fields in, so a save never clobbers config keys written
by a newer version (repo_update uses MERGE, which replaces the whole object).
config is only written as None when the merged result is genuinely empty.
(The previous approach relied on Pydantic extras, which ObjectModel's default
extra="ignore" silently dropped.)
- Update database/CLAUDE.md migration totals (15 up/down, incl. migration 15).
- Add tests for unmapped-key preservation and clearing num_ctx while keeping
other config keys.
* fix(credentials): validate config-mirrored extras via Pydantic
Address cubic re-review on #903: mirror known config keys (num_ctx) onto their
convenience fields in a `before` model validator so they go through normal
Pydantic field validation/coercion, instead of an `after`-validator
object.__setattr__ that bypassed type checks. Adds a test asserting num_ctx is
coerced to int and a non-coercible value is rejected.
* feat(podcasts): integrate model registry for profiles and credential passthrough
Replace loose provider/model string fields with record<model> references
in podcast profiles, enabling credential passthrough to podcast-creator.
Backend:
- EpisodeProfile: outline_llm, transcript_llm (record<model>) replace
outline_provider/outline_model strings. New language field (BCP 47).
- SpeakerProfile: voice_model (record<model>) replaces tts_provider/
tts_model strings. Per-speaker voice_model override support.
- Migration 14: schema changes making legacy fields optional, adding new
record<model> fields.
- Data migration (migration.py): auto-converts legacy profiles to model
registry references on startup. Idempotent.
- podcast_commands.py: resolves credentials for ALL profiles before
calling podcast-creator.
- New /api/languages endpoint (pycountry + babel) with BCP 47 locale
codes (pt-BR, en-US, etc.).
Frontend:
- Episode/speaker profile forms use ModelSelector instead of manual
provider/model dropdowns.
- Language dropdown with BCP 47 codes in episode profile form.
- Per-speaker TTS voice model override in speaker profile form.
- "Templates" tab renamed to "Profiles".
- Setup required badge on unconfigured profiles.
- i18n updated across all 8 locales.
Closes#486, closes#552
* fix(i18n): remove unused legacy podcast provider/model keys
Remove 10 orphaned i18n keys across all 8 locales that were left behind
after replacing manual provider/model dropdowns with ModelSelector.
* fix: address review violations in podcast model registry
- P1: Remove profiles with failed model resolution from dicts to prevent
podcast-creator validation errors on unrelated profiles
- P2: Use centralized QUERY_KEYS.languages instead of inline key
- P3: Fix ISO 639-1 → BCP 47 in model field description and CLAUDE.md
- P3: Update "templates" → "profiles" in locale string values (all 8)
* chore: bump version to 1.8.0
* feat: replace provider config with credential-based system (#477)
Introduce a new credential management system replacing the old
ProviderConfig singleton and standalone Models page. Each credential
stores encrypted API keys and provider-specific configuration with
full CRUD support via a unified settings UI.
Backend:
- Add Credential domain model with encrypted API key storage
- Add credentials API router (CRUD, discovery, registration, testing)
- Add encryption utilities for secure key storage
- Add key_provider for DB-first env-var fallback provisioning
- Add connection tester and model discovery services
- Integrate ModelManager with credential-based config
- Add provider name normalization for Esperanto compatibility
- Add database migrations 11-12 for credential schema
Frontend:
- Rewrite settings/api-keys page with credential management UI
- Add model discovery dialog with search and custom model support
- Add compact default model assignments (primary/advanced layout)
- Add inline model testing and credential connection testing
- Add env-var migration banner
- Update navigation to unified settings page
- Remove standalone models page and old settings components
i18n:
- Update all 7 locale files with credential and model management keys
Closes#477
Co-Authored-By: JFMD <git@jfmd.us>
Co-Authored-By: OraCatQAQ <570768706@qq.com>
* fix: address PR #540 review comments
- Fix docs referencing removed Models page
- Fix error-handler returning raw messages instead of i18n keys
- Fix auth.py misleading docstring and missing no-password guard
- Fix connection_tester using wrong env var for openai_compatible
- Add provision_provider_keys before model discovery/sync
- Update CLAUDE.md to reflect credential-based system
- Fix missing closing brace in api-keys page useEffect
* fix: add logging to credential migration and surface errors in UI
- Add comprehensive logging to migrate-from-env and
migrate-from-provider-config endpoints (start, per-provider
progress, success/failure with stack traces, final summary)
- Fix frontend migration hooks ignoring errors array from response
- Show error toast when migration fails instead of "nothing to migrate"
- Invalidate status/envStatus queries after migration so banner updates
* docs: update CLAUDE.md files for credential system
Replace stale ProviderConfig and /api-keys/ references across 8 CLAUDE.md
files to reflect the new Credential-based system from PR #540.
* docs: update user documentation for credential-based system
Replace env var API key instructions with Settings UI credential
workflow across all user-facing documentation. The new flow is:
set OPEN_NOTEBOOK_ENCRYPTION_KEY → start services → add credential
in Settings UI → test → discover models → register.
- Rewrite ai-providers.md, api-configuration.md, environment-reference.md
- Update all quick-start guides and installation docs
- Update ollama.md, openai-compatible.md, local-tts/stt networking sections
- Update reverse-proxy.md, development-setup.md, security.md
- Fix broken links to non-existent docs/deployment/ paths
- Add credentials endpoints to api-reference.md
- Move all API key env vars to deprecated/legacy sections
* chore: bump version to 1.7.0-rc1
Release candidate for credential-based provider management system.
* fix: initialize provider before try block in test_credential
Prevents UnboundLocalError when Credential.get() throws (e.g.,
invalid credential_id) before provider is assigned.
* fix: reorder down migration to drop index before table
Removes duplicate REMOVE FIELD statement and reorders so the index
is dropped before the table, preventing rollback failures.
* refactor: simplify encryption key to always derive via SHA-256
Remove the dual code path in _ensure_fernet_key() that detected native
Fernet keys. Since the credential system is new, always deriving via
SHA-256 removes unnecessary complexity. Also removes the generate_key()
function and Fernet.generate_key() references from docs.
* fix: correct mock patch targets in embedding tests and URL validation
Fix embedding tests patching wrong module path for model_manager
(was targeting open_notebook.utils.embedding.model_manager but it's
imported locally from open_notebook.ai.models). Also fix URL validation
to allow unresolvable hostnames since they may be valid in the
deployment environment (e.g., Azure endpoints, internal DNS).
* feat: add global setup banner for encryption and migration status
Show a persistent banner in AppShell when encryption key is missing
(red) or env var API keys can be migrated (amber), so users see
these prompts on every page instead of only on Settings > API Keys.
Includes a docs link for the encryption banner and i18n support
across all 7 locales.
* docs: several improvements to docker-compose e env examples
* Update README.md
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
* docs: fix env var format in README and update model setup instructions
Align the encryption key snippet in README Step 2 with the list
format used in the compose file. Replace deprecated "Settings →
Models" instructions with credential-based Discover Models flow.
* fix: address credential system review issues
- Fix SSRF bypass via IPv4-mapped IPv6 addresses (::ffff:169.254.x.x)
- Fix TTS connection test missing config parameter
- Add Azure-specific model discovery using api-key auth header
- Add Vertex static model list for credential-based discovery
- Fix PROVIDER_DISCOVERY_FUNCTIONS incorrect azure/vertex mapping
- Extract business logic to api/credentials_service.py (service layer)
- Move credential Pydantic schemas to api/models.py
- Update tests to use new service imports and ValueError assertions
* fix: sanitize error responses and migrate key_provider to Credential
- Replace raw exception messages in all credential router 500 responses
with generic error strings (internal details logged server-side only)
- Refactor key_provider.py to use Credential.get_by_provider() instead
of deprecated ProviderConfig.get_instance()
- Remove unused functions (get_provider_configs, get_default_api_key,
get_provider_config) that were dead code
---------
Co-authored-by: JFMD <git@jfmd.us>
Co-authored-by: OraCatQAQ <570768706@qq.com>
* fix(i18n): resolve podcast dialog translation infinite loop and profile issues
- Remove incorrect translation keys for user-defined episode profiles
- Cache translation strings in ContentSelectionPanel to avoid repeated
Proxy accesses that triggered infinite loop detection
- Stabilize useEffect dependencies with dataKey pattern to prevent
re-initialization on every keystroke
- Replace unstable sourcesQueries prop with stable fetchingNotebookIds set
- Clean up unused getSourceModes function and TranslationKeys import
* chore: bump lock
* chore: bump version to 1.5.1 and update CHANGELOG
* fix: guard .join() call in dataKey when query data is undefined
* fix(api): use FETCH command instead of async status lookups for sources list
Replace N async calls to surreal-commands with SurrealDB FETCH clause
to resolve command status in a single query. This eliminates the
command status cascade bottleneck.
* perf(db): add indexes on source field for insights and embeddings
Add migration #10 that creates indexes on the `source` field of
`source_insight` and `source_embedding` tables. These indexes
dramatically improve the performance of source listing queries
that use subqueries to count insights and check embedding existence.
Performance improvement: ~8.5s -> ~0.3s for 30 sources (28x faster)
* perf(db): make index concurrent
* fix: add IF NOT EXISTS to index definitions for idempotency
* fix: address code review feedback
- Add IF EXISTS to rollback migration for safer rollbacks
- Add fallback for unresolved command references (status = "unknown")
New front-end
Launch Chat API
Manage Sources
Enable re-embedding of all contents
Sources can be added without a notebook now
Improved settings
Enable model selector on all chats
Background processing for better experience
Dark mode
Improved Notes
Improved Docs:
- Remove all Streamlit references from documentation
- Update deployment guides with React frontend setup
- Fix Docker environment variables format (SURREAL_URL, SURREAL_PASSWORD)
- Update docker image tag from :latest to :v1-latest
- Change navigation references (Settings → Models to just Models)
- Update development setup to include frontend npm commands
- Add MIGRATION.md guide for users upgrading from Streamlit
- Update quick-start guide with correct environment variables
- Add port 5055 documentation for API access
- Update project structure to reflect frontend/ directory
- Remove outdated source-chat documentation files
Creates the API layer for Open Notebook
Creates a services API gateway for the Streamlit front-end
Migrates the SurrealDB SDK to the official one
Change all database calls to async
New podcast framework supporting multiple speaker configurations
Implement the surreal-commands library for async processing
Improve docker image and docker-compose configurations
