* fix(sources): fall back to auto when a selected engine's runtime is absent
The content-processing engine choice is persisted in the database; the
runtime that serves it (Docling, local Crawl4AI) is installed on demand
from environment flags evaluated at boot. The two therefore drift: a
redeploy that drops OPEN_NOTEBOOK_ENABLE_CRAWL4AI/_DOCLING, a volume
moved to a new deployment, or a failed on-demand install all leave a
stored selection pointing at a runtime that is not there.
The source graph passed that selection straight to content-core, so
every affected extraction failed with "Could not extract any text
content from this source" - no mention of the engine, the runtime, or
the flag that would fix it. For a URL engine set to crawl4ai this breaks
URL ingestion entirely.
The graph now checks runtime availability before honoring the stored
engine and degrades to content-core's "auto" chain, logging a WARNING
that names the engine and the env var that would enable it. Engines with
no opt-in runtime (auto/simple/firecrawl/jina) are passed through
untouched.
The availability probes moved from api/routers/capabilities.py to
open_notebook/utils/runtime_capabilities.py so the graph can use them
without importing from the API layer; the capabilities endpoint keeps
identical behavior and its tests follow the probes to their new home.
Found by the smoke-e2e agent during v1.14.0 release testing, on a dev
environment that was in exactly this state. Pre-existing since v1.13.0
(#1122 made the runtimes opt-in, #432 made the stored selection take
effect), not a v1.14.0 regression.
* docs(changelog): record the unavailable-engine fallback fix
The DNS-pinning guard added in #1063 is the highest-risk change in the
v1.14.0 release for self-hosters: it sits on the outbound path of every
credential save, connection test and model discovery, so a regression
that over-blocks silently costs users their provider.
TestPinnedHttpTarget already covers what the guard must reject. This
adds the inverse assertions - the deployment shapes that must keep
working - which prepare_pinned_http_target had no coverage for at all
(the existing legitimate-use tests only exercised validate_url, which
is not what gates outbound requests):
- Ollama on localhost, including IPv4 preference when both families
resolve
- host.docker.internal (containerized app -> host service)
- private LAN by IP literal (LM Studio) and by hostname
- IPv6 loopback literal
- Tailscale CGNAT space (100.64.0.0/10) - shared, not link-local
- AAAA-only endpoints producing a bracketed, parseable URL
- query strings surviving the rewrite (PPQ's ?type=all discovery URL)
All pass against the current implementation - this pins the behavior
rather than fixing a defect.
Surface content-core 2.x's docling_formulas (formula extraction) and
docling_vision (image/chart vision) enrichment flags in Settings →
Content Processing, mirroring the existing OCR toggle. Both default off
and are gated on Docling availability in the UI. The settings persist
via GET/PUT /api/settings and are threaded into content-core extraction
alongside docling_ocr. Migration 23 backfills the new fields on the
existing content_settings record. Labels and help are translated across
all 14 locales.
Closes#1131
* fix(proxy): keep internal SurrealDB websocket out of HTTP proxy (#1160)
websockets 15.0 auto-detects HTTP_PROXY/HTTPS_PROXY and tunnels even
ws:// connections through the proxy. The SurrealDB SDK connects over a
websocket, so with a proxy set the internal DB connection was routed
through the external proxy, which rejected the internal host with HTTP
403 and killed the worker/API on startup.
- Add ensure_internal_no_proxy() helper that merges host.docker.internal,
surrealdb, localhost, 127.0.0.1 into no_proxy/NO_PROXY (never clobbering
a user value) and call it at API, worker and DB-module startup.
- Add host.docker.internal and surrealdb to the .env.example and docs
NO_PROXY examples.
- Add unit tests for the injection helper.
* fix(proxy): preserve NO_PROXY wildcard and include custom SurrealDB host (#1160)
Address review findings on the no_proxy injection:
- NO_PROXY=* (bypass all hosts) is now treated as terminal: leave the
user's config untouched instead of narrowing the wildcard to a finite
list by appending the internal hosts.
- Parse the SurrealDB host from SURREAL_URL (falling back to
SURREAL_ADDRESS) and add it to the bypass list, so deployments with a
custom DB host/IP no longer route DB traffic through the proxy. Unset
or malformed values fall back to the four defaults gracefully.
- Drop the inaccurate getproxies() caching remark in the test.
* fix(models): stop auto-assign from re-filling cleared optional defaults
Auto-assign treated every empty default slot as "missing" and filled it,
so an optional slot a user deliberately cleared (to fall back to the chat
model) got silently re-populated on the next run, undoing the intent.
- Auto-assign now fills only the required slots (chat, embedding); the
optional slots (transformation, tools, large context, TTS, STT) are
left untouched.
- get_default_model("large_context") now falls back to the chat model
when unset, matching transformation/tools (TTS/STT still return None).
- Settings UI shows an inline hint on each empty optional slot: the text
slots show "using chat model (<name>)"; TTS/STT show a not-configured
hint. Required slots remain non-clearable. New i18n keys across all 14
locales.
Closes#1098
* test: guard await_args against None for mypy
PPQ's bare /v1/models returns only chat/language models; ?type=all is
required to also list the embedding, STT and TTS models it advertises as
a multi-modality gateway. Without it those modalities never surfaced in
discovery. PPQ_MODEL_TYPES already classifies the returned ids.
Closes#1180
esperanto 2.25.1 forwards a custom base_url for anthropic_compatible
models natively in to_langchain(), so the ChatAnthropic re-injection
shim added by #1043 is no longer needed.
- provision.py: drop _to_langchain() helper; call model.to_langchain()
- models.py: drop the _open_notebook_provider marker (shim-only consumer)
- tests: assert base_url forwarding via the native to_langchain() path
Closes#1055
Register anthropic_compatible in the provider registry so its env config,
modalities, test model, and /api/providers entry are derived from PROVIDERS
(#1075's single source of truth); the only manual copy is the SupportedProvider
Literal. Maps to esperanto's anthropic provider with a custom base_url, and
re-injects that base_url via ChatAnthropic since esperanto's to_langchain drops
it. Connection-test and model discovery mirror the openai_compatible siblings,
including DNS-rebinding pinning (prepare_pinned_http_target). A single shared
validator enforces the base_url + api_key requirement on both the create and
update paths.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Expose Esperanto's built-in omlx OpenAICompatibleProfile in Settings with
Ollama-style UX (default http://localhost:11435/v1, optional API key,
language+embedding discovery). No openai-compatible remapping or
OPENAI_COMPATIBLE_* env mirroring.
Fixes#1048
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Wire four esperanto 2.25.1 providers into the provider matrix:
- Cohere (COHERE_API_KEY): language + embedding via the native v2 API,
with a bespoke discoverer (AIFactory.get_provider_models) since Cohere
is not OpenAI-compatible. Reranking is out of scope (#1087).
- Deepgram: extend modalities to add speech_to_text (Nova/Whisper)
alongside the existing Aura text_to_speech voices.
- PayPerQ / PPQ (PPQ_API_KEY): multi-modality OpenAI-compatible gateway
(https://api.ppq.ai/v1), auto-discovered via the /models endpoint.
- Novita (NOVITA_API_KEY): OpenAI-compatible LLM gateway
(https://api.novita.ai/openai), auto-discovered via /models.
Registry-derived surfaces (env config, modalities, test models, discovery
table, GET /api/providers) update automatically; the SupportedProvider
Literal, key_provider config, availability env map, docs and tests were
updated to match. Closes#1170
esperanto 2.25.0 added OpenRouter for TTS and STT (previously LLM/embedding
only). Expose those modalities in Open Notebook.
- provider_registry: openrouter modalities -> all four (adds speech_to_text,
text_to_speech)
- model_discovery: bespoke discover_openrouter_models combines the live
OpenAI-compatible /models listing with a static seed of the audio model ids
esperanto ships as defaults (microsoft/mai-voice-2 for TTS, openai/whisper-1
and openai/whisper-large-v3 for STT), since OpenRouter's listing does not
reliably tag audio models. Seed only when live discovery returns models, so a
failed discovery never registers unusable audio models.
- credentials_service: credential-based discovery seeds the same audio ids for
openrouter (only after a successful /models response)
- connection_tester: exclude openrouter from DEFAULT_TEST_VOICES so TTS tests
use esperanto's model-specific available_voices (voices are model-specific;
the default model uses Microsoft neural voice names, not alloy/nova)
- tests, docs (provider matrix, AI provider guides) and CHANGELOG
Closes#987
Credential.to_esperanto_config() emitted the generic project/location
keys for all providers, but esperanto's Vertex providers accept
vertex_project/vertex_location. Credential-linked (non-env) Vertex TTS
therefore crashed with "__init__() got an unexpected keyword argument
'project'". Map to the Vertex-specific key names for the vertex provider
only; the Credential schema/API fields stay project/location and
non-Vertex providers are unaffected.
Closes#1151
* fix(notebook): cascade-delete chat sessions on notebook deletion
Deleting a notebook removed its notes and exclusive sources but left
chat_session records orphaned. Extend Notebook.delete() to enumerate the
notebook's chat sessions via the existing refers_to relation and delete
each one, and report chat_session_count in the delete preview.
Closes#1124
* refactor(notebook): drop chat-session count from delete preview
The delete-preview API advertised chat_session_count but the frontend
dialog and locales never render it, creating a UI inconsistency. Trim
the preview back to notes and sources; the deletion cascade and its
post-delete deleted_chat_sessions count are unchanged.
Each transformation persists its own model_id, but neither call site that
invokes the transformation graph passed a config, so run_transformation()
always read a None model and fell back to the global default.
Forward transformation.model_id through the LangGraph `configurable` config
at both call sites (source-processing graph and the run_transformation
background command). An unset model_id remains None and falls back to the
default via the existing branch in provision_langchain_model().
Closes#1137
* fix(sources): cap error text surfaced to clients
Source processing status (get_source_status) and sync-processing failures
returned the raw command/result error_message to clients unbounded, which
could leak arbitrary internal exception text and didn't match the error
capping applied elsewhere in the API.
Add a None-safe _truncate_error helper (200-char cap, ellipsis when cut)
and apply it on both paths. Adds focused unit tests for the helper.
Closes#1136
* test: narrow Optional return before assertions to satisfy mypy
* fix: pin DNS for outbound provider HTTP requests
Close the DNS-rebinding TOCTOU left by validate_url alone by resolving
once and connecting to the vetted IP (Host/SNI preserved). Apply
consistently to openai_compatible, ollama, and azure discovery/test paths.
* fix: restore @dataclass newline in model_discovery
* fix: resolve mypy failure and pin openai custom base_url discovery
- Cast getaddrinfo sockaddr[0] to str in _resolve_safe_ips so the
list[str] append typechecks (mypy CI gate was failing).
- Apply DNS pinning to the openai provider's user-supplied base_url
discovery path, mirroring the openai_compatible path, so httpx cannot
re-resolve to a metadata address after validation (DNS-rebinding TOCTOU).
- Add a test asserting the pinned IP URL, Host header and SNI extension
reach httpx on the openai base_url discovery path.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Docling and local Crawl4AI are now opt-in, installed on first container startup
via OPEN_NOTEBOOK_ENABLE_DOCLING / OPEN_NOTEBOOK_ENABLE_CRAWL4AI, keeping the
default image lean (Crawl4AI un-bundled). Downloads cache on the /app/data
volume; installs are blocking with loud logs and degrade-don't-die on failure.
A GET /api/capabilities probe reports actual availability and the Settings UI
gates the Docling/Crawl4AI engines and OCR toggle (with env-var hints) until the
runtime is present. Docs folded from #1121 and rewritten to opt-in; ADR-007 added.
Closes#1122. Closes#1105.
Expose content-core's docling_ocr flag as a user setting. OCR runs on
scanned PDFs and images when the Docling engine handles them; it's on by
default (matching content-core) and can be turned off for faster
processing of text-native documents.
- New `docling_ocr` boolean on ContentSettings (default True), plumbed
through the settings API and into ContentCoreConfig in the source graph.
- "Enable OCR" checkbox in the Content Processing settings card, with
label + help translated across all 14 locales.
Part of #939. Stacked on #432.
* feat(sources): add Crawl4AI URL engine and honor persisted engine settings (#432)
- Add "crawl4ai" as a selectable URL processing engine (domain Literal,
settings API validation, SettingsForm select, and label across all 14
locales; urlHelp updated in en-US to describe the new fallback chain).
- The source graph now loads the persisted ContentSettings and passes the
document/URL engine choices to ContentCoreConfig. Previously it built a
hard-coded ContentSettings with "auto" engines, so a user's selection in
Settings never took effect. Falls back to defaults if settings can't load.
- Crawl4AI Docker mode is driven by content-core's native CRAWL4AI_API_URL
env var (documented separately under #1105).
Part of #939.
* fix(432): bundle Crawl4AI runtime + address review
- Bundle the Crawl4AI runtime so its local, no-API-key mode works out of
the box: depend on content-core[crawl4ai] and install the Chromium
browser via playwright in the Docker runtime-base (both image variants).
Footprint is modest (no torch/transformers/CUDA); image grows ~300 MB
from Chromium + system libs.
- Preserve the server-side traceback when persisted content settings fail
to load (logger.opt(exception=True)) instead of only the message.
- Reset the ContentSettings singleton between domain tests (clear_instance)
so a non-default value can't leak into neighboring tests.
Addresses review on #432.
* i18n(432): translate urlHelp Crawl4AI description across all 13 non-en locales
The Crawl4AI engine label was already localized; this brings the URL-engine
'help me choose' text in line with en-US in every locale — describing
Crawl4AI (local JS rendering, no API key) and its place in the auto
fallback chain (Firecrawl -> Jina -> Crawl4AI -> simple).
Unsupported files used to enqueue a background job that failed and then
burned the full 15-attempt retry budget (~1h) before showing a generic
"Failed" with no actionable detail.
- Add a pre-flight `_assert_file_supported()` using content-core 2.x's
header-only `check_file_support()` (same routing as real extraction) in
the upload branch of `_build_content_state`, before any job is enqueued
and before the source record is created.
- Guard the source-retry endpoint the same way. Unexpected check errors
(e.g. file removed before a retry) fall through to normal extraction.
- Map `UnsupportedTypeException` to `415 Unsupported Media Type` via a
dedicated global handler (previously fell through to the base
OpenNotebookError handler's 500). The message names the detected type.
Part of #939.
* chore(sources): upgrade to content-core 2.x
Bump content-core 1.14.x -> 2.0.4 and adapt the source graph to the new
keyword-only extract_content API.
- extract_content is now keyword-only; engine/model overrides move to
ContentCoreConfig (audio model still sourced from Default Models).
- ProcessSourceState was removed; the graph now consumes ExtractionOutput,
which no longer echoes url/file_path back — carry those from the input
state into the saved Asset.
- content-core 2.x no longer deletes the uploaded source file, so honor
delete_source on our side after a successful extraction.
- drop the obsolete output_format param (markdown is the default).
pymupdf (AGPL) is replaced transitively by pdfplumber (MIT); moviepy is
gone (direct ffmpeg), which fixes MP3-with-chapters audio.
Part of #939.
* docs(changelog): note content-core 2.x upgrade (#1103)
* fix(sources): wire YouTube transcript language preferences into ContentCoreConfig
content-core's default youtube_languages is only en/es/pt. Pass the broader
list Open Notebook has always intended so non-English videos still resolve a
transcript. (cubic P2 on #1116)
Since #1112 dropped the legacy provider/model strings, EpisodeCard's outline,
transcript and speaker rows showed "— / —" for new episodes. The API now
resolves the snapshot's model references (outline_llm/transcript_llm/voice_model)
to provider/name display fields at serialization time, batched into a single
query per request via Model.get_display_info_for_ids so listing episodes never
does a per-row model lookup. The card falls back to the legacy snapshot strings
for old episodes and degrades to a dash when a referenced model was deleted.
Closes#1114
Migration 22 best-effort maps profiles whose outline_llm/transcript_llm/
voice_model references are still empty to existing model records
(provider + name + type, no auto-create), clears the legacy values and
drops the 6 columns. The startup data migration that retried this
mapping on every boot (open_notebook/podcasts/migration.py) is deleted
along with its api/main.py lifespan hook; the legacy fields are removed
from the Pydantic models, API schemas and frontend types/panels.
Accepted trade-off: profiles whose mapping never converged lose the
legacy strings and stay unresolved - they were already non-functional
and the UI already flags them; the user re-picks models once.
* refactor(podcasts): store audio paths relative to PODCASTS_FOLDER
audio_file previously stored the absolute path returned by
podcast-creator (sometimes as a file:// URI), which kept path traversal
representable in the DB and broke playback when DATA_FOLDER moved.
- New single choke point open_notebook/podcasts/audio_paths.py:
to_relative_audio_path() validates at write time (the DB can never
hold an absolute or root-escaping value) and
resolve_contained_audio_path() joins + resolves + contains at read
time, replacing the per-router guards from #1018. Absolute/file://
legacy values are treated as invalid (same 403/404 as before).
- Generation command stores the relative form; in-band "ERROR: ..."
values from podcast-creator now fail the job with the real error.
- build_episode_output_dir() builds from PODCASTS_FOLDER so the write
root and the validation root cannot drift.
- Migration 21 converts legacy rows under the known roots (plain
file:/// URIs, /app/data/podcasts/, /data/podcasts/,
./data/podcasts/, data/podcasts/) via a single-strip IF/ELSE chain;
rows under other roots stay untouched by design.
Closes#1030
* test: use the tolerant migration-count assertion pattern
Per review: assert >= 21 with up/down parity and a distinctive-SQL check on
index 20, matching the convention from the migration-19/20 tests, so the next
migration doesn't trip an unrelated test file.
* fix(podcasts): reference speaker profiles by record ID instead of name
episode_profile.speaker_config now stores a record<speaker_profile>
reference (migration 20 converts existing rows; orphaned names become
null). The generate API keeps accepting the speaker profile by name and
resolves it to a record ID at the boundary.
Closes#630
* fix(podcasts): survive orphaned speaker references in podcast-creator config
Review fixes: rewrite record IDs back to speaker names (and drop orphaned
profiles) when building podcast-creator's episode config so one orphaned
profile can't fail validation for every generation; clearer error for
dangling references; resolve a single speaker name on single-record
endpoints instead of fetching the whole speaker table; accurate wording
about record links not preventing dangling references.
* fix(podcasts): submit the speaker profile record ID from the generate dialog
Submitting the cached display name made generation fail if the speaker
profile was renamed while the dialog was open - the record ID is stable
and the API resolves either form.
The settings frontend now fetches the provider list from the backend
registry endpoint (session-cached react-query hook useProviders())
instead of keeping its own hardcoded copies of provider names, display
names, modalities and docs URLs in lib/providers.tsx.
- New api module (lib/api/providers.ts) + hook (lib/hooks/use-providers.ts)
with staleTime: Infinity — the list only changes on deploy.
- lib/providers.tsx reduced to modality presentation (icon/color/label)
behind fallback-safe helpers, so an unknown modality from a future
provider still renders instead of breaking.
- The backend registry declaration order is the display order (verified
identical to the old curated ALL_PROVIDERS order; endpoint test now
pins order, not just set-equality).
- api-keys page gains loading/error states for the provider fetch; new
i18n keys added to all 14 locales.
- Deleted the regex-based frontend/backend sync test; the
SupportedProvider Literal test remains the backend guarantee.
- Renamed the dead useProviders() in use-models.ts (availability
endpoint) to useProviderAvailability() to avoid a name collision.
- Updated stale docs/comments (AGENTS.md, credentials.md, cubic.yaml,
provider_registry.py, api/models.py) that still described the frontend
table as a manual sync point.
Closes#1082
source_insight was the only content table without created/updated field
definitions. Since the table is SCHEMAFULL and insights are created via a
raw CREATE (create_insight command), SurrealDB silently dropped any
timestamps - rows genuinely held NONE - and the API then wrapped them in
str(), so clients received the literal string "None".
- Migration 19 defines created/updated on source_insight with the same
time::now() defaults used by source, note and notebook, so new insights
are stamped at creation. Existing rows are left untouched (no backfill).
- SourceInsightResponse.created/updated are now Optional[str]; both
routers emit an ISO 8601 string when the timestamp is present and null
when it is absent (legacy rows), never "None".
- Frontend types updated to string | null accordingly.
- New tests cover the migration definition/registration and the API
serialization (absent -> null, present -> ISO string).
Note: Python-side stamping alone was not viable - the table is SCHEMAFULL,
so undefined fields written by the client are silently dropped on
SurrealDB 2.x (verified against a live instance), hence the schema
migration mirroring the other tables.
Closes#1045
PUT /api/models/defaults used 'is not None' guards, so an explicit null
sent to clear a default was silently ignored — the old value survived
while the client saw success (same anti-pattern fixed for credentials
in #1046). The handler is now keyed on field presence (model_fields_set):
absent keeps the current value, explicit null clears it. The required
defaults (chat, embedding) reject null with a 400.
In the UI, the optional default-model selects (transformation, tools,
large context, TTS, STT) now offer a clear option — labeled 'Use
fallback (chat default)' for transformation/tools, which fall back to
the chat default when unset, and 'None' otherwise — and display that
state when no model is assigned. The transformation default is no
longer marked required, matching the backend fallback behavior.
Closes#1091
The command always re-derived the speaker from
episode_profile.speaker_config, silently ignoring the speaker_profile
value accepted by POST /api/podcasts/generate. Resolve the explicitly
requested profile when provided and fall back to the episode profile's
speaker_config otherwise. speaker_profile is now Optional on
PodcastGenerationInput so the fallback is a real contract.
Closes#1044
The valid-provider gate test posted a real credential through the route;
with a developer .env loaded (SURREAL_URL + encryption key set) every
pytest run persisted a 'Test' openai credential to the live dev database.
Force the encryption-key gate to fail so the request proves validation
clearance without ever reaching persistence.
Consolidate the three copies of context assembly into
open_notebook/utils/context_builder.py:
- POST /api/chat/context now delegates to build_notebook_context()
(same request/response shapes, same string-matching config semantics)
- The source-chat graph now calls build_source_context() instead of the
495-line generalized ContextBuilder class, which had exactly one
caller and whose notebook/notes/priority-config flexibility was dead
- POST /api/notebooks/{notebook_id}/context removed: it duplicated
/api/chat/context with a slightly different envelope and had zero
callers (frontend, docs, tests)
Behavior is pinned by new characterization tests written before the
refactor (tests/test_context_endpoint_characterization.py) plus unit
tests for build_source_context.
The routers wrapped endpoint bodies in a broad 'except Exception' that
re-raised everything as HTTPException(500), intercepting the typed
open_notebook.exceptions hierarchy before the global handlers in
api/main.py could map it to its documented status code (NotFoundError
-> 404, InvalidInputError -> 400, ConfigurationError -> 422,
RateLimitError -> 429, NetworkError/ExternalServiceError -> 502).
Every endpoint-level handler chain across the 18 affected routers now
re-raises HTTPException and OpenNotebookError before the final generic
arm, which keeps catching untyped exceptions and returning a sanitized
500 (the no-raw-exception-text guarantee is unchanged and covered by
tests/test_error_message_sanitization.py). Nested best-effort blocks
(fallbacks, SSE error events, per-item loops) are deliberately left
untouched. In create_source and _create_source_async_path the new
re-raise arms preserve the uploaded-file / orphaned-source cleanup.
Characterization tests that pinned the old wrapped-500 behavior for
source-chat sessions with a missing refers_to relation are updated to
the correct 404, and a new test module asserts per router that a
domain-layer ConfigurationError maps to 422 instead of 500.
* ci: gate PRs on mypy, start ignore_errors burn-down
Add a backend-typecheck CI job running uv run python -m mypy . and bring
the repo-wide baseline from 197 errors to 0 so the gate blocks new type
errors from now on:
- enable the pydantic mypy plugin (resolves 138 false positives on
models whose fields have Field(None, ...) defaults)
- fix the remaining errors with real annotations; the only new type:
ignore comments cover a genuine langgraph typing limitation (partial
state dicts are valid at runtime but the overloads require the full
state type) and tests that intentionally pass invalid input
- start the ignore_errors burn-down: open_notebook.graphs.transformation,
open_notebook.graphs.ask and api.routers.models are now type-checked;
stale blocks for the deleted api.client and api.podcast_api_service
modules removed. Only open_notebook.domain.notebook remains exempt
(DB layer is migrating to surreal-basics)
- fix the mypy.ini header comment to describe what the config does
* test: use typing.get_args on the registry literal check
The registry test landed in parallel using Literal.__args__, which the
mypy gate in this branch rejects; align it with the get_args() idiom
used by the rest of the file.
* refactor(ai): single provider registry as the backend source of truth
Provider metadata (env vars, modalities, connection-test models,
OpenAI-compatible discovery URLs, display names, docs links) is now
defined once in open_notebook/ai/provider_registry.py. The existing
surfaces are derived from it, keeping every import and call-site shape
unchanged:
- api/credentials_service.py: PROVIDER_ENV_CONFIG, PROVIDER_MODALITIES
and the discovery url_map are built from the registry
- open_notebook/ai/connection_tester.py: TEST_MODELS derived
- open_notebook/ai/model_discovery.py: OPENAI_COMPAT_PROVIDERS built
from registry entries with a discovery URL (quirk hooks stay local)
The SupportedProvider Literal (typing, can't be built at runtime) and
the frontend provider tables remain manual copies; the cross-check
tests now assert registry keys == Literal == frontend list, plus
registry internal consistency and discovery-table coverage.
New GET /api/providers endpoint exposes the registry (name, display
name, modalities, docs_url, env-configured status) so clients can stop
hardcoding provider lists (frontend adoption is a follow-up).
Docs updated: open_notebook/AGENTS.md and docs/7-DEVELOPMENT/credentials.md
now describe the registry instead of the four-place sync rule.
* refactor(ai): address review findings on the provider registry
- Build PROVIDERS via _build_registry(), which raises on a duplicate
provider name at import time instead of silently dropping the earlier
spec (dict-comprehension behavior); regression test added
- Pin the exact OpenAI-compatible provider -> discovery URL mapping in
a test so a registry edit can't silently drop or misassign a URL
- Give TEST_MODELS a real type annotation
(Dict[str, Tuple[Optional[str], str]]) instead of bare dict
* test(api): characterize shared chat/source-chat router behaviors
* refactor(api): extract shared session and message helpers for chat routers
* refactor(api): mark intentionally unused source id unpacks
The session handlers only need the verified session; the source-level
verification happens inside get_verified_source_session. Underscore the
unused binding in get/update/delete to make that explicit.
Collapse the eight structurally identical OpenAI-compatible provider
discovery functions (openai, groq, mistral, deepseek, xai, openrouter,
dashscope, minimax) into one generic discover_openai_compatible_provider()
driven by an OPENAI_COMPAT_PROVIDERS spec table. Per-provider quirks
(Mistral capability flags, OpenRouter fixed type + description) are kept
as optional hooks on the spec. Module-level function names and
PROVIDER_DISCOVERY_FUNCTIONS keys are preserved.
Replace the stale hardcoded Anthropic model list (claude-3 era) with
real discovery via GET https://api.anthropic.com/v1/models (paginated,
x-api-key + anthropic-version headers) — the comment claiming Anthropic
has no listing API was wrong. Keep a refreshed static fallback
(ANTHROPIC_FALLBACK_MODELS, current Claude 4.x/5 aliases) for when the
API call fails, and reuse the same fetch-with-fallback in the
credential-based discovery path in api/credentials_service.py, which
carried a second copy of the stale list.
Add tests covering the table-driven path, provider quirks, Anthropic
pagination/headers, and the fallback behavior (all HTTP mocked).
Remove F401 (unused imports), F841 (unused local variables) and E722
(bare except) from the ruff ignore list; the legacy Streamlit-era code
that motivated ignoring them is gone.
Fallout fixed: 10 unused imports removed (none were load-bearing
re-exports or side-effect imports), 2 unused mock bindings in tests
dropped, and a now-empty TYPE_CHECKING block in
open_notebook/utils/embedding.py cleaned up. No bare excepts remained
in the codebase.
The embed_single_item, embed_chunk and vectorize_source command handlers
existed only so jobs queued by a pre-1.6 version could drain after an
upgrade; any worker restarted on 1.6+ has no such jobs. Remove them, their
input/output models and their tests.
Also drop dead tooling config from pyproject.toml: the [tool.mypy] block
(mypy.ini takes precedence and is the real config) and the Streamlit-era
ruff per-file-ignores for app_home.py and pages/**, which no longer exist.
* fix(frontend): actually clear credential fields when emptied in the edit dialog
Clearing base_url (or the Vertex project/location/credentials_path)
in the credential edit dialog silently did nothing: the submit handler
mapped an emptied field to `undefined`, JSON.stringify dropped the key
from the PUT body, and the backend's partial-update semantics kept the
old value — while the UI reported success.
Emptied fields are now sent as an explicit `null`, which the API
already accepts and persists as a cleared value (verified live).
UpdateCredentialRequest's nullable fields are typed accordingly.
Found in v1.11 release testing (pre-existing, not a release regression):
an Ollama credential with a stale IP in base_url could not be cleared
from the UI.
* fix(api): clear credential fields on explicit null, not just empty string
The update handler guarded every field with 'is not None', so a JSON
null sent to clear base_url (or endpoint/api_version/endpoint_*/
project/location/credentials_path/num_ctx) was silently skipped — the
old value survived while the client saw a 200. Combined with the
frontend bug fixed in the previous commit (emptied fields dropped from
the payload entirely), clearing a credential field was impossible from
the UI: during release testing an Ollama credential kept pointing at a
stale IP after being 'cleared', and only an empty-string PUT crafted by
hand actually cleared it.
Field updates are now keyed on presence in model_fields_set: absent
keeps the old value, explicit null or "" clears. name/modalities/
api_key keep their non-null guards (clearing those is not meaningful).
Regression tests cover null-clears, empty-string-clears, absent-keeps
and the Vertex credentials_path case.
* test(frontend): extract credential update payload builder and cover clear-on-empty
Addresses the cubic review: the frontend half of the fix had no
reproducing test. The edit dialog's payload construction now lives in
a pure buildCredentialUpdatePayload() (frontend/src/lib/
credential-update-payload.ts) used by the component, with tests that
fail on the original bug: an emptied base_url (and Vertex project/
location/credentials_path) must survive JSON.stringify as an explicit
null, unchanged fields must be omitted, and emptied num_ctx clears
via 0.
Two findings from v1.11 release testing:
- GET /api/sources?sort_by=title returned a 500. source.title carries a
SEARCH (BM25) index (idx_source_title, migration 1) and SurrealDB's
planner fails ORDER BY on such a column with 'No iterator has been
found'. The query now sorts by a computed alias
(string::lowercase(title OR '') AS title_sort), which sidesteps the
index and makes the sort case-insensitive as a bonus.
- POST /api/sources with an over-limit notebooks/transformations array
(or invalid JSON in either field) returned a raw 500.
parse_source_form_data() builds SourceCreate manually, so pydantic's
ValidationError never reached FastAPI's request-validation handler.
Both cases now surface as a clean 422 with a descriptive message.
Verified against a live SurrealDB v2 instance; regression tests added
for the ORDER BY alias, all six sort fields, and the 422 paths.
* fix: make the provider connection test resilient to model retirement
Two independent hardenings against the #970 class of breakage (a hard-coded
Gemini test model getting shut down by Google, which made testing a valid
key fail with a 404):
- Use Google's floating alias gemini-flash-latest for the Google/Vertex
test model instead of a dated id, so a retirement repoints it for us.
- Reframe the provider connection test around what an error actually
proves: only a rejected key (401), missing permissions (403), or an
unreachable endpoint are failures. Anything the provider returns after
authenticating - a rate limit, or a missing/retired/unsupported model -
still proves the credentials work, so it reports success. Previously this
relied on matching the literal phrase 'not found' + 'model', which a
differently-worded retirement/deprecation error slipped past.
Unifies the auth/network/rate-limit classification (previously duplicated
and divergent between connection_tester and credentials_service) into
shared helpers. The individual-model test keeps model-not-found as a
failure, since there a specific registered model really is broken.
Adds classification tests with realistic provider error strings.
* docs: add CHANGELOG entry for connection-test resilience fix
* fix: update Google model version in connection tester and tests
* fix: updated the gemini model lists that references deprecated models
* fix: update also documentation. This is more prone to not follow the maintainer's directives, so PTADL
* fix: forgot two references to gemini deprecated versions
* fix: use valid, longer-lived gemini model IDs
Several IDs the PR introduced don't exist or are near shutdown, verified
against Google's official model/deprecation pages (2026-07):
- connection test model gemini-2.5-flash -> gemini-3.5-flash (2.5-flash
retires 2026-10-16; 3.5-flash is the current stable GA and Google's
named replacement, so the #970 fix doesn't re-break in ~3 months)
- gemini-3.5-pro (does not exist) -> gemini-2.5-pro in docs
- plain gemini-3.1-flash (not a GA Gemini-API id) dropped from the Vertex
discovery list / preferences / docs; use gemini-3.5-flash or -flash-lite
- dead gemini-pro dropped from preferences and docs
Ported #996's #970 regression test, pinned to gemini-3.5-flash.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: prevent SurrealQL injection in Credential.get_all()'s order_by
ObjectModel.get_all() already validates order_by against an allowlist
before interpolating it into the query string - but Credential overrides
get_all() (to handle per-row api_key decryption) and its override built
the ORDER BY clause with a raw, unvalidated f-string, bypassing that
protection entirely.
Extract the base class's validation logic into a reusable
_validate_order_by() classmethod and route Credential.get_all() through
it. Not reachable from any current API surface today (the only caller
passes a hardcoded "provider, created", not user input), but any
subclass that builds its own query around order_by instead of
delegating to the base get_all() needs this so the allowlist can't
silently drift between call sites.
No test currently exercises this path - worth adding one (e.g.
Credential.get_all(order_by="field; DROP TABLE credential") raises
InvalidInputError) before merging.
* test: add order_by injection regression tests
Covers _validate_order_by() normalization/rejection and the Credential
.get_all() path, which raises on an injected clause before any query
reaches the database.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: surface silent command-submission failures where they matter
Source.add_insight() caught submission failures and returned None
instead of raising - callers (transformation.py, source.py) run inside
surreal-commands jobs whose outer exception handling already
retries/fails on this, so a swallowed submission failure meant a
transformation could report success while the insight was silently
never persisted. Now raises DatabaseOperationError, matching
vectorize()'s existing contract.
Note.save()'s auto-embed needs the opposite treatment: it's an implicit
side effect of save() (not an explicit dedicated call), and the note
itself is already durably saved by the time it runs - so a submission
hiccup there shouldn't turn an otherwise-successful save into a 500.
Wrapped in try/except, logs and returns None. api/routers/embedding.py's
explicit POST /embed (item_type=note) is the one caller for whom
submission success genuinely is the point of the call, so it separately
checks for a missing command_id and surfaces that as a failure.
* fix: collapse duplicate error logging in add_insight
logger.error + logger.exception logged the same failure twice; a single
logger.exception call carries both the message and the traceback (same
form vectorize() already uses).
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Two small fixes to the podcast episode-listing path, plus a doc note:
- audio_file is only ever set server-side today from a UUID-named
directory under PODCASTS_FOLDER, so this can't currently be tripped -
but the stream/retry/delete endpoints didn't verify the resolved path
actually stayed within PODCASTS_FOLDER before following it. Add
_is_audio_path_contained() as defense in depth against a future code
path (e.g. importing external audio) setting audio_file to something
else.
- Listing episodes called get_job_detail() -> get_command_status() once
per episode, each its own round trip (no connection pooling). Add
PodcastEpisode.get_job_details_for_commands() to batch-fetch status
for every episode's command in one query instead.
Also documents (docs/7-DEVELOPMENT/security.md) that podcast_creator's
configure("templates", {...}) compiles strings as Jinja2 template
source - the same shape as the SSTI vulnerability fixed in
transformation.py (GHSA-f35w-wx37-26q7). Confirmed dormant: no code path
in this repo calls it today. commands/podcast_commands.py gets a
matching code comment warning against wiring user text into it if a
"custom podcast template" feature is ever added.
api/routers/sources.py and api/podcast_service.py interpolated the raw
exception (detail=f"...: {str(e)}") into client-facing error responses -
inconsistent with the safer pattern already used elsewhere in the same
files (e.g. the download handlers), which log the raw exception
server-side but return a fixed generic message. Internal details (DB
hostnames, connection errors, stack-trace fragments) could leak to any
API caller through a 500 response.
Every occurrence already had a matching logger.error() call, so this is
a client-facing message change only, not a logging change. Deliberate
app-authored messages (InvalidInputError text, "Notebook X not found",
result.error_message) are untouched - only raw f"{str(e)}" interpolation
is affected.
tests/test_config_endpoint_no_leak.py is a regression lock for
api/routers/config.py (already correct, not touched by this diff) rather
than a fix - added since it shares the same "unauthenticated endpoint,
don't leak exception text" concern.
* fix: restrict CreateCredentialRequest.provider to a known allowlist
provider was a bare `str`, so any string (typo'd or bogus) flowed
through validation to the domain layer and failed later with a less
clear error, instead of a clean 422 at the API boundary.
Add a SupportedProvider Literal covering the 17 providers already
handled elsewhere - kept in sync with the frontend's ALL_PROVIDERS,
connection_tester.py's TEST_MODELS, and credentials_service.py's
PROVIDER_ENV_CONFIG (a test asserts all three agree on the same set).
* test: lock the frontend provider list into the sync test
The frontend's ALL_PROVIDERS copy was only cross-checked in a docstring;
extract its string literals from the source so all four provider lists
(Literal, TEST_MODELS, PROVIDER_ENV_CONFIG, frontend) are enforced by CI.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Three small, independent hardening fixes to source ingestion:
- generate_unique_filename() checked `if not resolved.exists()` then let
a separate write happen later - two concurrent uploads landing on the
same candidate name could both pass the check and clobber each other.
Now atomically claims the name via Path.touch(exist_ok=False) (O_EXCL)
as part of the search loop itself.
- _resolve_source_file() and _is_source_file_available() compared
`resolved_path.startswith(safe_root)` without a trailing separator - a
sibling directory that merely starts with the same string (e.g.
"uploads_evil/") would incorrectly be treated as contained, unlike this
file's other two path checks which already guard with `+ os.sep`. Not
reachable today (source.asset.file_path is only ever set server-side),
but this closes the gap and matches the existing pattern.
- SourceCreate.notebooks/transformations had no length limit; both are
iterated with a per-item DB lookup in create_source(), so an unbounded
array let a single request trigger an unbounded number of sequential DB
round trips. Capped at 50.
tests/test_upload_type_mitigations.py adds no code change - it documents
why the adjacent "no file type allowlist on uploads" finding was
investigated and judged low-risk without one (downloads are already
served as application/octet-stream regardless of actual file type).
