* fix(sources): fall back to auto when a selected engine's runtime is absent
The content-processing engine choice is persisted in the database; the
runtime that serves it (Docling, local Crawl4AI) is installed on demand
from environment flags evaluated at boot. The two therefore drift: a
redeploy that drops OPEN_NOTEBOOK_ENABLE_CRAWL4AI/_DOCLING, a volume
moved to a new deployment, or a failed on-demand install all leave a
stored selection pointing at a runtime that is not there.
The source graph passed that selection straight to content-core, so
every affected extraction failed with "Could not extract any text
content from this source" - no mention of the engine, the runtime, or
the flag that would fix it. For a URL engine set to crawl4ai this breaks
URL ingestion entirely.
The graph now checks runtime availability before honoring the stored
engine and degrades to content-core's "auto" chain, logging a WARNING
that names the engine and the env var that would enable it. Engines with
no opt-in runtime (auto/simple/firecrawl/jina) are passed through
untouched.
The availability probes moved from api/routers/capabilities.py to
open_notebook/utils/runtime_capabilities.py so the graph can use them
without importing from the API layer; the capabilities endpoint keeps
identical behavior and its tests follow the probes to their new home.
Found by the smoke-e2e agent during v1.14.0 release testing, on a dev
environment that was in exactly this state. Pre-existing since v1.13.0
(#1122 made the runtimes opt-in, #432 made the stored selection take
effect), not a v1.14.0 regression.
* docs(changelog): record the unavailable-engine fallback fix
Surface content-core 2.x's docling_formulas (formula extraction) and
docling_vision (image/chart vision) enrichment flags in Settings →
Content Processing, mirroring the existing OCR toggle. Both default off
and are gated on Docling availability in the UI. The settings persist
via GET/PUT /api/settings and are threaded into content-core extraction
alongside docling_ocr. Migration 23 backfills the new fields on the
existing content_settings record. Labels and help are translated across
all 14 locales.
Closes#1131
* fix(proxy): keep internal SurrealDB websocket out of HTTP proxy (#1160)
websockets 15.0 auto-detects HTTP_PROXY/HTTPS_PROXY and tunnels even
ws:// connections through the proxy. The SurrealDB SDK connects over a
websocket, so with a proxy set the internal DB connection was routed
through the external proxy, which rejected the internal host with HTTP
403 and killed the worker/API on startup.
- Add ensure_internal_no_proxy() helper that merges host.docker.internal,
surrealdb, localhost, 127.0.0.1 into no_proxy/NO_PROXY (never clobbering
a user value) and call it at API, worker and DB-module startup.
- Add host.docker.internal and surrealdb to the .env.example and docs
NO_PROXY examples.
- Add unit tests for the injection helper.
* fix(proxy): preserve NO_PROXY wildcard and include custom SurrealDB host (#1160)
Address review findings on the no_proxy injection:
- NO_PROXY=* (bypass all hosts) is now treated as terminal: leave the
user's config untouched instead of narrowing the wildcard to a finite
list by appending the internal hosts.
- Parse the SurrealDB host from SURREAL_URL (falling back to
SURREAL_ADDRESS) and add it to the bypass list, so deployments with a
custom DB host/IP no longer route DB traffic through the proxy. Unset
or malformed values fall back to the four defaults gracefully.
- Drop the inaccurate getproxies() caching remark in the test.
* fix(models): stop auto-assign from re-filling cleared optional defaults
Auto-assign treated every empty default slot as "missing" and filled it,
so an optional slot a user deliberately cleared (to fall back to the chat
model) got silently re-populated on the next run, undoing the intent.
- Auto-assign now fills only the required slots (chat, embedding); the
optional slots (transformation, tools, large context, TTS, STT) are
left untouched.
- get_default_model("large_context") now falls back to the chat model
when unset, matching transformation/tools (TTS/STT still return None).
- Settings UI shows an inline hint on each empty optional slot: the text
slots show "using chat model (<name>)"; TTS/STT show a not-configured
hint. Required slots remain non-clearable. New i18n keys across all 14
locales.
Closes#1098
* test: guard await_args against None for mypy
Register anthropic_compatible in the provider registry so its env config,
modalities, test model, and /api/providers entry are derived from PROVIDERS
(#1075's single source of truth); the only manual copy is the SupportedProvider
Literal. Maps to esperanto's anthropic provider with a custom base_url, and
re-injects that base_url via ChatAnthropic since esperanto's to_langchain drops
it. Connection-test and model discovery mirror the openai_compatible siblings,
including DNS-rebinding pinning (prepare_pinned_http_target). A single shared
validator enforces the base_url + api_key requirement on both the create and
update paths.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Expose Esperanto's built-in omlx OpenAICompatibleProfile in Settings with
Ollama-style UX (default http://localhost:11435/v1, optional API key,
language+embedding discovery). No openai-compatible remapping or
OPENAI_COMPATIBLE_* env mirroring.
Fixes#1048
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Wire four esperanto 2.25.1 providers into the provider matrix:
- Cohere (COHERE_API_KEY): language + embedding via the native v2 API,
with a bespoke discoverer (AIFactory.get_provider_models) since Cohere
is not OpenAI-compatible. Reranking is out of scope (#1087).
- Deepgram: extend modalities to add speech_to_text (Nova/Whisper)
alongside the existing Aura text_to_speech voices.
- PayPerQ / PPQ (PPQ_API_KEY): multi-modality OpenAI-compatible gateway
(https://api.ppq.ai/v1), auto-discovered via the /models endpoint.
- Novita (NOVITA_API_KEY): OpenAI-compatible LLM gateway
(https://api.novita.ai/openai), auto-discovered via /models.
Registry-derived surfaces (env config, modalities, test models, discovery
table, GET /api/providers) update automatically; the SupportedProvider
Literal, key_provider config, availability env map, docs and tests were
updated to match. Closes#1170
esperanto 2.25.0 added OpenRouter for TTS and STT (previously LLM/embedding
only). Expose those modalities in Open Notebook.
- provider_registry: openrouter modalities -> all four (adds speech_to_text,
text_to_speech)
- model_discovery: bespoke discover_openrouter_models combines the live
OpenAI-compatible /models listing with a static seed of the audio model ids
esperanto ships as defaults (microsoft/mai-voice-2 for TTS, openai/whisper-1
and openai/whisper-large-v3 for STT), since OpenRouter's listing does not
reliably tag audio models. Seed only when live discovery returns models, so a
failed discovery never registers unusable audio models.
- credentials_service: credential-based discovery seeds the same audio ids for
openrouter (only after a successful /models response)
- connection_tester: exclude openrouter from DEFAULT_TEST_VOICES so TTS tests
use esperanto's model-specific available_voices (voices are model-specific;
the default model uses Microsoft neural voice names, not alloy/nova)
- tests, docs (provider matrix, AI provider guides) and CHANGELOG
Closes#987
* fix(notebook): cascade-delete chat sessions on notebook deletion
Deleting a notebook removed its notes and exclusive sources but left
chat_session records orphaned. Extend Notebook.delete() to enumerate the
notebook's chat sessions via the existing refers_to relation and delete
each one, and report chat_session_count in the delete preview.
Closes#1124
* refactor(notebook): drop chat-session count from delete preview
The delete-preview API advertised chat_session_count but the frontend
dialog and locales never render it, creating a UI inconsistency. Trim
the preview back to notes and sources; the deletion cascade and its
post-delete deleted_chat_sessions count are unchanged.
* fix(sources): cap error text surfaced to clients
Source processing status (get_source_status) and sync-processing failures
returned the raw command/result error_message to clients unbounded, which
could leak arbitrary internal exception text and didn't match the error
capping applied elsewhere in the API.
Add a None-safe _truncate_error helper (200-char cap, ellipsis when cut)
and apply it on both paths. Adds focused unit tests for the helper.
Closes#1136
* test: narrow Optional return before assertions to satisfy mypy
* fix: pin DNS for outbound provider HTTP requests
Close the DNS-rebinding TOCTOU left by validate_url alone by resolving
once and connecting to the vetted IP (Host/SNI preserved). Apply
consistently to openai_compatible, ollama, and azure discovery/test paths.
* fix: restore @dataclass newline in model_discovery
* fix: resolve mypy failure and pin openai custom base_url discovery
- Cast getaddrinfo sockaddr[0] to str in _resolve_safe_ips so the
list[str] append typechecks (mypy CI gate was failing).
- Apply DNS pinning to the openai provider's user-supplied base_url
discovery path, mirroring the openai_compatible path, so httpx cannot
re-resolve to a metadata address after validation (DNS-rebinding TOCTOU).
- Add a test asserting the pinned IP URL, Host header and SNI extension
reach httpx on the openai base_url discovery path.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Docling and local Crawl4AI are now opt-in, installed on first container startup
via OPEN_NOTEBOOK_ENABLE_DOCLING / OPEN_NOTEBOOK_ENABLE_CRAWL4AI, keeping the
default image lean (Crawl4AI un-bundled). Downloads cache on the /app/data
volume; installs are blocking with loud logs and degrade-don't-die on failure.
A GET /api/capabilities probe reports actual availability and the Settings UI
gates the Docling/Crawl4AI engines and OCR toggle (with env-var hints) until the
runtime is present. Docs folded from #1121 and rewritten to opt-in; ADR-007 added.
Closes#1122. Closes#1105.
Expose content-core's docling_ocr flag as a user setting. OCR runs on
scanned PDFs and images when the Docling engine handles them; it's on by
default (matching content-core) and can be turned off for faster
processing of text-native documents.
- New `docling_ocr` boolean on ContentSettings (default True), plumbed
through the settings API and into ContentCoreConfig in the source graph.
- "Enable OCR" checkbox in the Content Processing settings card, with
label + help translated across all 14 locales.
Part of #939. Stacked on #432.
* feat(sources): add Crawl4AI URL engine and honor persisted engine settings (#432)
- Add "crawl4ai" as a selectable URL processing engine (domain Literal,
settings API validation, SettingsForm select, and label across all 14
locales; urlHelp updated in en-US to describe the new fallback chain).
- The source graph now loads the persisted ContentSettings and passes the
document/URL engine choices to ContentCoreConfig. Previously it built a
hard-coded ContentSettings with "auto" engines, so a user's selection in
Settings never took effect. Falls back to defaults if settings can't load.
- Crawl4AI Docker mode is driven by content-core's native CRAWL4AI_API_URL
env var (documented separately under #1105).
Part of #939.
* fix(432): bundle Crawl4AI runtime + address review
- Bundle the Crawl4AI runtime so its local, no-API-key mode works out of
the box: depend on content-core[crawl4ai] and install the Chromium
browser via playwright in the Docker runtime-base (both image variants).
Footprint is modest (no torch/transformers/CUDA); image grows ~300 MB
from Chromium + system libs.
- Preserve the server-side traceback when persisted content settings fail
to load (logger.opt(exception=True)) instead of only the message.
- Reset the ContentSettings singleton between domain tests (clear_instance)
so a non-default value can't leak into neighboring tests.
Addresses review on #432.
* i18n(432): translate urlHelp Crawl4AI description across all 13 non-en locales
The Crawl4AI engine label was already localized; this brings the URL-engine
'help me choose' text in line with en-US in every locale — describing
Crawl4AI (local JS rendering, no API key) and its place in the auto
fallback chain (Firecrawl -> Jina -> Crawl4AI -> simple).
Unsupported files used to enqueue a background job that failed and then
burned the full 15-attempt retry budget (~1h) before showing a generic
"Failed" with no actionable detail.
- Add a pre-flight `_assert_file_supported()` using content-core 2.x's
header-only `check_file_support()` (same routing as real extraction) in
the upload branch of `_build_content_state`, before any job is enqueued
and before the source record is created.
- Guard the source-retry endpoint the same way. Unexpected check errors
(e.g. file removed before a retry) fall through to normal extraction.
- Map `UnsupportedTypeException` to `415 Unsupported Media Type` via a
dedicated global handler (previously fell through to the base
OpenNotebookError handler's 500). The message names the detected type.
Part of #939.
Since #1112 dropped the legacy provider/model strings, EpisodeCard's outline,
transcript and speaker rows showed "— / —" for new episodes. The API now
resolves the snapshot's model references (outline_llm/transcript_llm/voice_model)
to provider/name display fields at serialization time, batched into a single
query per request via Model.get_display_info_for_ids so listing episodes never
does a per-row model lookup. The card falls back to the legacy snapshot strings
for old episodes and degrades to a dash when a referenced model was deleted.
Closes#1114
Migration 22 best-effort maps profiles whose outline_llm/transcript_llm/
voice_model references are still empty to existing model records
(provider + name + type, no auto-create), clears the legacy values and
drops the 6 columns. The startup data migration that retried this
mapping on every boot (open_notebook/podcasts/migration.py) is deleted
along with its api/main.py lifespan hook; the legacy fields are removed
from the Pydantic models, API schemas and frontend types/panels.
Accepted trade-off: profiles whose mapping never converged lose the
legacy strings and stay unresolved - they were already non-functional
and the UI already flags them; the user re-picks models once.
* refactor(podcasts): store audio paths relative to PODCASTS_FOLDER
audio_file previously stored the absolute path returned by
podcast-creator (sometimes as a file:// URI), which kept path traversal
representable in the DB and broke playback when DATA_FOLDER moved.
- New single choke point open_notebook/podcasts/audio_paths.py:
to_relative_audio_path() validates at write time (the DB can never
hold an absolute or root-escaping value) and
resolve_contained_audio_path() joins + resolves + contains at read
time, replacing the per-router guards from #1018. Absolute/file://
legacy values are treated as invalid (same 403/404 as before).
- Generation command stores the relative form; in-band "ERROR: ..."
values from podcast-creator now fail the job with the real error.
- build_episode_output_dir() builds from PODCASTS_FOLDER so the write
root and the validation root cannot drift.
- Migration 21 converts legacy rows under the known roots (plain
file:/// URIs, /app/data/podcasts/, /data/podcasts/,
./data/podcasts/, data/podcasts/) via a single-strip IF/ELSE chain;
rows under other roots stay untouched by design.
Closes#1030
* test: use the tolerant migration-count assertion pattern
Per review: assert >= 21 with up/down parity and a distinctive-SQL check on
index 20, matching the convention from the migration-19/20 tests, so the next
migration doesn't trip an unrelated test file.
* fix(podcasts): reference speaker profiles by record ID instead of name
episode_profile.speaker_config now stores a record<speaker_profile>
reference (migration 20 converts existing rows; orphaned names become
null). The generate API keeps accepting the speaker profile by name and
resolves it to a record ID at the boundary.
Closes#630
* fix(podcasts): survive orphaned speaker references in podcast-creator config
Review fixes: rewrite record IDs back to speaker names (and drop orphaned
profiles) when building podcast-creator's episode config so one orphaned
profile can't fail validation for every generation; clearer error for
dangling references; resolve a single speaker name on single-record
endpoints instead of fetching the whole speaker table; accurate wording
about record links not preventing dangling references.
* fix(podcasts): submit the speaker profile record ID from the generate dialog
Submitting the cached display name made generation fail if the speaker
profile was renamed while the dialog was open - the record ID is stable
and the API resolves either form.
The settings frontend now fetches the provider list from the backend
registry endpoint (session-cached react-query hook useProviders())
instead of keeping its own hardcoded copies of provider names, display
names, modalities and docs URLs in lib/providers.tsx.
- New api module (lib/api/providers.ts) + hook (lib/hooks/use-providers.ts)
with staleTime: Infinity — the list only changes on deploy.
- lib/providers.tsx reduced to modality presentation (icon/color/label)
behind fallback-safe helpers, so an unknown modality from a future
provider still renders instead of breaking.
- The backend registry declaration order is the display order (verified
identical to the old curated ALL_PROVIDERS order; endpoint test now
pins order, not just set-equality).
- api-keys page gains loading/error states for the provider fetch; new
i18n keys added to all 14 locales.
- Deleted the regex-based frontend/backend sync test; the
SupportedProvider Literal test remains the backend guarantee.
- Renamed the dead useProviders() in use-models.ts (availability
endpoint) to useProviderAvailability() to avoid a name collision.
- Updated stale docs/comments (AGENTS.md, credentials.md, cubic.yaml,
provider_registry.py, api/models.py) that still described the frontend
table as a manual sync point.
Closes#1082
source_insight was the only content table without created/updated field
definitions. Since the table is SCHEMAFULL and insights are created via a
raw CREATE (create_insight command), SurrealDB silently dropped any
timestamps - rows genuinely held NONE - and the API then wrapped them in
str(), so clients received the literal string "None".
- Migration 19 defines created/updated on source_insight with the same
time::now() defaults used by source, note and notebook, so new insights
are stamped at creation. Existing rows are left untouched (no backfill).
- SourceInsightResponse.created/updated are now Optional[str]; both
routers emit an ISO 8601 string when the timestamp is present and null
when it is absent (legacy rows), never "None".
- Frontend types updated to string | null accordingly.
- New tests cover the migration definition/registration and the API
serialization (absent -> null, present -> ISO string).
Note: Python-side stamping alone was not viable - the table is SCHEMAFULL,
so undefined fields written by the client are silently dropped on
SurrealDB 2.x (verified against a live instance), hence the schema
migration mirroring the other tables.
Closes#1045
PUT /api/models/defaults used 'is not None' guards, so an explicit null
sent to clear a default was silently ignored — the old value survived
while the client saw success (same anti-pattern fixed for credentials
in #1046). The handler is now keyed on field presence (model_fields_set):
absent keeps the current value, explicit null clears it. The required
defaults (chat, embedding) reject null with a 400.
In the UI, the optional default-model selects (transformation, tools,
large context, TTS, STT) now offer a clear option — labeled 'Use
fallback (chat default)' for transformation/tools, which fall back to
the chat default when unset, and 'None' otherwise — and display that
state when no model is assigned. The transformation default is no
longer marked required, matching the backend fallback behavior.
Closes#1091
Consolidate the three copies of context assembly into
open_notebook/utils/context_builder.py:
- POST /api/chat/context now delegates to build_notebook_context()
(same request/response shapes, same string-matching config semantics)
- The source-chat graph now calls build_source_context() instead of the
495-line generalized ContextBuilder class, which had exactly one
caller and whose notebook/notes/priority-config flexibility was dead
- POST /api/notebooks/{notebook_id}/context removed: it duplicated
/api/chat/context with a slightly different envelope and had zero
callers (frontend, docs, tests)
Behavior is pinned by new characterization tests written before the
refactor (tests/test_context_endpoint_characterization.py) plus unit
tests for build_source_context.
The routers wrapped endpoint bodies in a broad 'except Exception' that
re-raised everything as HTTPException(500), intercepting the typed
open_notebook.exceptions hierarchy before the global handlers in
api/main.py could map it to its documented status code (NotFoundError
-> 404, InvalidInputError -> 400, ConfigurationError -> 422,
RateLimitError -> 429, NetworkError/ExternalServiceError -> 502).
Every endpoint-level handler chain across the 18 affected routers now
re-raises HTTPException and OpenNotebookError before the final generic
arm, which keeps catching untyped exceptions and returning a sanitized
500 (the no-raw-exception-text guarantee is unchanged and covered by
tests/test_error_message_sanitization.py). Nested best-effort blocks
(fallbacks, SSE error events, per-item loops) are deliberately left
untouched. In create_source and _create_source_async_path the new
re-raise arms preserve the uploaded-file / orphaned-source cleanup.
Characterization tests that pinned the old wrapped-500 behavior for
source-chat sessions with a missing refers_to relation are updated to
the correct 404, and a new test module asserts per router that a
domain-layer ConfigurationError maps to 422 instead of 500.
* ci: gate PRs on mypy, start ignore_errors burn-down
Add a backend-typecheck CI job running uv run python -m mypy . and bring
the repo-wide baseline from 197 errors to 0 so the gate blocks new type
errors from now on:
- enable the pydantic mypy plugin (resolves 138 false positives on
models whose fields have Field(None, ...) defaults)
- fix the remaining errors with real annotations; the only new type:
ignore comments cover a genuine langgraph typing limitation (partial
state dicts are valid at runtime but the overloads require the full
state type) and tests that intentionally pass invalid input
- start the ignore_errors burn-down: open_notebook.graphs.transformation,
open_notebook.graphs.ask and api.routers.models are now type-checked;
stale blocks for the deleted api.client and api.podcast_api_service
modules removed. Only open_notebook.domain.notebook remains exempt
(DB layer is migrating to surreal-basics)
- fix the mypy.ini header comment to describe what the config does
* test: use typing.get_args on the registry literal check
The registry test landed in parallel using Literal.__args__, which the
mypy gate in this branch rejects; align it with the get_args() idiom
used by the rest of the file.
* refactor(ai): single provider registry as the backend source of truth
Provider metadata (env vars, modalities, connection-test models,
OpenAI-compatible discovery URLs, display names, docs links) is now
defined once in open_notebook/ai/provider_registry.py. The existing
surfaces are derived from it, keeping every import and call-site shape
unchanged:
- api/credentials_service.py: PROVIDER_ENV_CONFIG, PROVIDER_MODALITIES
and the discovery url_map are built from the registry
- open_notebook/ai/connection_tester.py: TEST_MODELS derived
- open_notebook/ai/model_discovery.py: OPENAI_COMPAT_PROVIDERS built
from registry entries with a discovery URL (quirk hooks stay local)
The SupportedProvider Literal (typing, can't be built at runtime) and
the frontend provider tables remain manual copies; the cross-check
tests now assert registry keys == Literal == frontend list, plus
registry internal consistency and discovery-table coverage.
New GET /api/providers endpoint exposes the registry (name, display
name, modalities, docs_url, env-configured status) so clients can stop
hardcoding provider lists (frontend adoption is a follow-up).
Docs updated: open_notebook/AGENTS.md and docs/7-DEVELOPMENT/credentials.md
now describe the registry instead of the four-place sync rule.
* refactor(ai): address review findings on the provider registry
- Build PROVIDERS via _build_registry(), which raises on a duplicate
provider name at import time instead of silently dropping the earlier
spec (dict-comprehension behavior); regression test added
- Pin the exact OpenAI-compatible provider -> discovery URL mapping in
a test so a registry edit can't silently drop or misassign a URL
- Give TEST_MODELS a real type annotation
(Dict[str, Tuple[Optional[str], str]]) instead of bare dict
- Extract _source_to_response() shared builder (SourceResponse was
hand-rolled 5x with the same nested AssetModel ternary)
- Extract _cleanup_uploaded_file() (the unlink ritual was pasted 6x
inside create_source); inner cleanups were redundant with the outer
exception handlers, which always run for the same failures
- Split create_source into _build_content_state() (type validation +
SSRF/LFI guards) and _create_source_async_path() /
_create_source_sync_path()
- Unify the paginated list query (with/without notebook filter differed
only in the FROM clause and bound params)
Pure structural refactor: same status codes, error messages and
response shapes. Security checks (atomic filename claim via
touch(exist_ok=False), path-traversal containment, SSRF/LFI guards)
preserved verbatim, with their comments.
* test(api): characterize shared chat/source-chat router behaviors
* refactor(api): extract shared session and message helpers for chat routers
* refactor(api): mark intentionally unused source id unpacks
The session handlers only need the verified session; the source-level
verification happens inside get_verified_source_session. Underscore the
unused binding in get/update/delete to make that explicit.
Collapse the eight structurally identical OpenAI-compatible provider
discovery functions (openai, groq, mistral, deepseek, xai, openrouter,
dashscope, minimax) into one generic discover_openai_compatible_provider()
driven by an OPENAI_COMPAT_PROVIDERS spec table. Per-provider quirks
(Mistral capability flags, OpenRouter fixed type + description) are kept
as optional hooks on the spec. Module-level function names and
PROVIDER_DISCOVERY_FUNCTIONS keys are preserved.
Replace the stale hardcoded Anthropic model list (claude-3 era) with
real discovery via GET https://api.anthropic.com/v1/models (paginated,
x-api-key + anthropic-version headers) — the comment claiming Anthropic
has no listing API was wrong. Keep a refreshed static fallback
(ANTHROPIC_FALLBACK_MODELS, current Claude 4.x/5 aliases) for when the
API call fails, and reuse the same fetch-with-fallback in the
credential-based discovery path in api/credentials_service.py, which
carried a second copy of the stale list.
Add tests covering the table-driven path, provider quirks, Anthropic
pagination/headers, and the fallback behavior (all HTTP mocked).
Remove F401 (unused imports), F841 (unused local variables) and E722
(bare except) from the ruff ignore list; the legacy Streamlit-era code
that motivated ignoring them is gone.
Fallout fixed: 10 unused imports removed (none were load-bearing
re-exports or side-effect imports), 2 unused mock bindings in tests
dropped, and a now-empty TYPE_CHECKING block in
open_notebook/utils/embedding.py cleaned up. No bare excepts remained
in the codebase.
Delete api/client.py (synchronous httpx client calling the app's own
FastAPI server, a leftover from the removed Streamlit UI) and the 13
api/*_service.py wrappers around it. None of these files had any
importer in routers, commands or tests, verified by grepping the whole
repo for each module name.
Also remove commands/example_commands.py (process_text/analyze_data
demo commands from the surreal-commands README) and its export from
commands/__init__.py, and update remaining docstring examples to
reference the real generate_podcast command.
The real services (command_service, credentials_service,
podcast_service) are untouched.
* fix(frontend): actually clear credential fields when emptied in the edit dialog
Clearing base_url (or the Vertex project/location/credentials_path)
in the credential edit dialog silently did nothing: the submit handler
mapped an emptied field to `undefined`, JSON.stringify dropped the key
from the PUT body, and the backend's partial-update semantics kept the
old value — while the UI reported success.
Emptied fields are now sent as an explicit `null`, which the API
already accepts and persists as a cleared value (verified live).
UpdateCredentialRequest's nullable fields are typed accordingly.
Found in v1.11 release testing (pre-existing, not a release regression):
an Ollama credential with a stale IP in base_url could not be cleared
from the UI.
* fix(api): clear credential fields on explicit null, not just empty string
The update handler guarded every field with 'is not None', so a JSON
null sent to clear base_url (or endpoint/api_version/endpoint_*/
project/location/credentials_path/num_ctx) was silently skipped — the
old value survived while the client saw a 200. Combined with the
frontend bug fixed in the previous commit (emptied fields dropped from
the payload entirely), clearing a credential field was impossible from
the UI: during release testing an Ollama credential kept pointing at a
stale IP after being 'cleared', and only an empty-string PUT crafted by
hand actually cleared it.
Field updates are now keyed on presence in model_fields_set: absent
keeps the old value, explicit null or "" clears. name/modalities/
api_key keep their non-null guards (clearing those is not meaningful).
Regression tests cover null-clears, empty-string-clears, absent-keeps
and the Vertex credentials_path case.
* test(frontend): extract credential update payload builder and cover clear-on-empty
Addresses the cubic review: the frontend half of the fix had no
reproducing test. The edit dialog's payload construction now lives in
a pure buildCredentialUpdatePayload() (frontend/src/lib/
credential-update-payload.ts) used by the component, with tests that
fail on the original bug: an emptied base_url (and Vertex project/
location/credentials_path) must survive JSON.stringify as an explicit
null, unchanged fields must be omitted, and emptied num_ctx clears
via 0.
Two findings from v1.11 release testing:
- GET /api/sources?sort_by=title returned a 500. source.title carries a
SEARCH (BM25) index (idx_source_title, migration 1) and SurrealDB's
planner fails ORDER BY on such a column with 'No iterator has been
found'. The query now sorts by a computed alias
(string::lowercase(title OR '') AS title_sort), which sidesteps the
index and makes the sort case-insensitive as a bonus.
- POST /api/sources with an over-limit notebooks/transformations array
(or invalid JSON in either field) returned a raw 500.
parse_source_form_data() builds SourceCreate manually, so pydantic's
ValidationError never reached FastAPI's request-validation handler.
Both cases now surface as a clean 422 with a descriptive message.
Verified against a live SurrealDB v2 instance; regression tests added
for the ORDER BY alias, all six sort fields, and the 422 paths.
* fix: make the provider connection test resilient to model retirement
Two independent hardenings against the #970 class of breakage (a hard-coded
Gemini test model getting shut down by Google, which made testing a valid
key fail with a 404):
- Use Google's floating alias gemini-flash-latest for the Google/Vertex
test model instead of a dated id, so a retirement repoints it for us.
- Reframe the provider connection test around what an error actually
proves: only a rejected key (401), missing permissions (403), or an
unreachable endpoint are failures. Anything the provider returns after
authenticating - a rate limit, or a missing/retired/unsupported model -
still proves the credentials work, so it reports success. Previously this
relied on matching the literal phrase 'not found' + 'model', which a
differently-worded retirement/deprecation error slipped past.
Unifies the auth/network/rate-limit classification (previously duplicated
and divergent between connection_tester and credentials_service) into
shared helpers. The individual-model test keeps model-not-found as a
failure, since there a specific registered model really is broken.
Adds classification tests with realistic provider error strings.
* docs: add CHANGELOG entry for connection-test resilience fix
* fix: update Google model version in connection tester and tests
* fix: updated the gemini model lists that references deprecated models
* fix: update also documentation. This is more prone to not follow the maintainer's directives, so PTADL
* fix: forgot two references to gemini deprecated versions
* fix: use valid, longer-lived gemini model IDs
Several IDs the PR introduced don't exist or are near shutdown, verified
against Google's official model/deprecation pages (2026-07):
- connection test model gemini-2.5-flash -> gemini-3.5-flash (2.5-flash
retires 2026-10-16; 3.5-flash is the current stable GA and Google's
named replacement, so the #970 fix doesn't re-break in ~3 months)
- gemini-3.5-pro (does not exist) -> gemini-2.5-pro in docs
- plain gemini-3.1-flash (not a GA Gemini-API id) dropped from the Vertex
discovery list / preferences / docs; use gemini-3.5-flash or -flash-lite
- dead gemini-pro dropped from preferences and docs
Ported #996's #970 regression test, pinned to gemini-3.5-flash.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
check_api_password() (an unused HTTPBearer-based dependency, superseded
by PasswordAuthMiddleware) and its now-unused imports are dead code -
nothing calls it. Removed.
Docs across api/CLAUDE.md, docs/3-USER-GUIDE/api-configuration.md,
docs/5-CONFIGURATION/security.md, docs/7-DEVELOPMENT/security.md, and
docs/SECURITY_REVIEW.md still described a hardcoded default password
("open-notebook-change-me") that PasswordAuthMiddleware doesn't actually
have - if OPEN_NOTEBOOK_PASSWORD is unset, auth is fully disabled
instead. Updated to match actual behavior.
* docs: restructure documentation around AGENTS.md, VISION.md and decision records
- Consolidate 17 CLAUDE.md files into 3 AGENTS.md (root, backend, frontend);
CLAUDE.md files become @AGENTS.md pointers
- Add VISION.md: product identity + current posture with horizon clusters
- Add docs/7-DEVELOPMENT/decisions/ with 4 retroactive ADRs and 2 PDRs
- Add 5 new engineering docs pages (credentials, content-processing,
podcasts, prompts, frontend) absorbing knowledge from removed CLAUDE.md
- Dismember TRIAGE.md: label taxonomy into maintainer-guide.md, product
jurisprudence into VISION.md, operator heuristics stay local (gitignored)
- Add AI-assisted/agent-generated PR guidelines to contributing.md
- Convert README.dev.md into a pointer after migrating its unique content
(make workflow matrix, Docker publishing, add-a-language playbook)
- Fix stale docs: migration path/format, provider count, locale list;
fix broken links (docs/index.md, PR template, CONFIGURATION.md)
* docs: fix README doc links and add markdown link check to CI
- Repoint 9 README links to pages that actually exist in docs/
- Replace literal (link) placeholder in maintainer-guide templates
- Add scripts/check_md_links.py validating relative links in tracked
markdown (skips URLs, anchors and code spans)
- Add docs-links workflow running the check on PRs that touch markdown
* docs: add documentation restructure to changelog
* feat: add cubic.yaml with project-aware AI review agents
Three custom review agents (vision & principles alignment backed by
VISION.md, known mechanical caveats, security & testability), PR-contract
review instructions, and automatic ultrareviews for auth, credential,
encryption and migration changes.
* docs: graduate issue-first policy by change size
Small obvious fixes (typos, docs, tiny bugs, i18n completions) no longer
require an issue; features and architecture changes still do. Sizeable
PRs opened without an issue convert to draft while the issue goes
through triage (1-2 days). Applied consistently across contributing
guide, root CONTRIBUTING pointer, PR template, maintainer guide red
flags and cubic review instructions.
* docs: align PR template Related Issue section with graduated issue-first policy
* docs: address review — generalize ADR-002/004, unwrap hard-wrapped lines
- ADR-002 now records the general delegation rule (platform/media support
that needs heavy coding lives in focused external libraries) covering
Esperanto, Content Core and podcast-creator
- ADR-004 now records the durable decision (long-running work runs on
background workers — heavy content, varied machine sizes, never lock
usage) with the queue technology as a swappable implementation detail
pending #381
- Remove mid-paragraph hard line wrapping from authored docs to match
repo convention (one line per paragraph)
* fix: address cubic review — stale doc facts, make dev/full targets, link checker query strings
- credentials.md: only PROVIDER_CONFIG exists as a map; Vertex/Azure/
OpenAI-compatible provisioning is inline in _provision_*() functions
- content-processing.md: correct ContextConfig priority weights
(source 100 > insight 75 > note 50)
- development-setup.md + Makefile: make dev/full pointed at root compose
files that don't exist; targets now use examples/docker-compose-dev.yml
and examples/docker-compose-full-local.yml with --project-directory .
- check_md_links.py: strip query strings before file-existence checks
* fix: surface silent command-submission failures where they matter
Source.add_insight() caught submission failures and returned None
instead of raising - callers (transformation.py, source.py) run inside
surreal-commands jobs whose outer exception handling already
retries/fails on this, so a swallowed submission failure meant a
transformation could report success while the insight was silently
never persisted. Now raises DatabaseOperationError, matching
vectorize()'s existing contract.
Note.save()'s auto-embed needs the opposite treatment: it's an implicit
side effect of save() (not an explicit dedicated call), and the note
itself is already durably saved by the time it runs - so a submission
hiccup there shouldn't turn an otherwise-successful save into a 500.
Wrapped in try/except, logs and returns None. api/routers/embedding.py's
explicit POST /embed (item_type=note) is the one caller for whom
submission success genuinely is the point of the call, so it separately
checks for a missing command_id and surfaces that as a failure.
* fix: collapse duplicate error logging in add_insight
logger.error + logger.exception logged the same failure twice; a single
logger.exception call carries both the message and the traceback (same
form vectorize() already uses).
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Two small fixes to the podcast episode-listing path, plus a doc note:
- audio_file is only ever set server-side today from a UUID-named
directory under PODCASTS_FOLDER, so this can't currently be tripped -
but the stream/retry/delete endpoints didn't verify the resolved path
actually stayed within PODCASTS_FOLDER before following it. Add
_is_audio_path_contained() as defense in depth against a future code
path (e.g. importing external audio) setting audio_file to something
else.
- Listing episodes called get_job_detail() -> get_command_status() once
per episode, each its own round trip (no connection pooling). Add
PodcastEpisode.get_job_details_for_commands() to batch-fetch status
for every episode's command in one query instead.
Also documents (docs/7-DEVELOPMENT/security.md) that podcast_creator's
configure("templates", {...}) compiles strings as Jinja2 template
source - the same shape as the SSTI vulnerability fixed in
transformation.py (GHSA-f35w-wx37-26q7). Confirmed dormant: no code path
in this repo calls it today. commands/podcast_commands.py gets a
matching code comment warning against wiring user text into it if a
"custom podcast template" feature is ever added.
api/routers/sources.py and api/podcast_service.py interpolated the raw
exception (detail=f"...: {str(e)}") into client-facing error responses -
inconsistent with the safer pattern already used elsewhere in the same
files (e.g. the download handlers), which log the raw exception
server-side but return a fixed generic message. Internal details (DB
hostnames, connection errors, stack-trace fragments) could leak to any
API caller through a 500 response.
Every occurrence already had a matching logger.error() call, so this is
a client-facing message change only, not a logging change. Deliberate
app-authored messages (InvalidInputError text, "Notebook X not found",
result.error_message) are untouched - only raw f"{str(e)}" interpolation
is affected.
tests/test_config_endpoint_no_leak.py is a regression lock for
api/routers/config.py (already correct, not touched by this diff) rather
than a fix - added since it shares the same "unauthenticated endpoint,
don't leak exception text" concern.
* fix: restrict CreateCredentialRequest.provider to a known allowlist
provider was a bare `str`, so any string (typo'd or bogus) flowed
through validation to the domain layer and failed later with a less
clear error, instead of a clean 422 at the API boundary.
Add a SupportedProvider Literal covering the 17 providers already
handled elsewhere - kept in sync with the frontend's ALL_PROVIDERS,
connection_tester.py's TEST_MODELS, and credentials_service.py's
PROVIDER_ENV_CONFIG (a test asserts all three agree on the same set).
* test: lock the frontend provider list into the sync test
The frontend's ALL_PROVIDERS copy was only cross-checked in a docstring;
extract its string literals from the source so all four provider lists
(Literal, TEST_MODELS, PROVIDER_ENV_CONFIG, frontend) are enforced by CI.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Three small, independent hardening fixes to source ingestion:
- generate_unique_filename() checked `if not resolved.exists()` then let
a separate write happen later - two concurrent uploads landing on the
same candidate name could both pass the check and clobber each other.
Now atomically claims the name via Path.touch(exist_ok=False) (O_EXCL)
as part of the search loop itself.
- _resolve_source_file() and _is_source_file_available() compared
`resolved_path.startswith(safe_root)` without a trailing separator - a
sibling directory that merely starts with the same string (e.g.
"uploads_evil/") would incorrectly be treated as contained, unlike this
file's other two path checks which already guard with `+ os.sep`. Not
reachable today (source.asset.file_path is only ever set server-side),
but this closes the gap and matches the existing pattern.
- SourceCreate.notebooks/transformations had no length limit; both are
iterated with a per-item DB lookup in create_source(), so an unbounded
array let a single request trigger an unbounded number of sequential DB
round trips. Capped at 50.
tests/test_upload_type_mitigations.py adds no code change - it documents
why the adjacent "no file type allowlist on uploads" finding was
investigated and judged low-risk without one (downloads are already
served as application/octet-stream regardless of actual file type).
* fix: reject oversized request bodies before auth/routing
No limit existed on request body size, so a single upload could exhaust
memory/disk before any validation ran. MaxBodySizeMiddleware
(api/middleware.py) rejects requests over OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB
(default 100MB) via both a Content-Length pre-check and by counting bytes
as the body streams in, so it also catches chunked requests with no
Content-Length header.
Registered after PasswordAuthMiddleware (wrapping it) so oversized
requests are rejected before spending any work on credential checks, and
inside CORSMiddleware so a rejected upload still gets CORS headers.
* fix: clamp non-positive size limits and log rejected requests
OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB=0 (or negative) made every request with a
body 413; fall back to the default instead, with a warning. Also log each
413 rejection so operators can diagnose failed uploads.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: don't combine wildcard CORS origins with allow_credentials
Combining allow_origins=["*"] with allow_credentials=True makes
Starlette's CORSMiddleware reflect the request's Origin header verbatim
instead of returning a literal "*" (browsers reject a literal wildcard
alongside credentials) - defeating the origin allowlist for any
credentialed request.
allow_credentials is now tied to whether CORS_ORIGINS was explicitly
scoped: False for the default wildcard, True once an operator opts into
specific origins. _cors_headers() (the manual CORS builder for error
responses raised before CORSMiddleware runs) is updated to match, so it
can't grant credentials the real middleware wouldn't.
Not independently exploitable today (the frontend never sends
credentialed requests, and auth is a Bearer header, not a cookie), but
there's no reason to allow it for the default wildcard case.
* fix: key allow_credentials on the parsed origins list, not the env var
An operator who explicitly sets CORS_ORIGINS=* got allow_credentials=True
with a wildcard origin list - the exact reflect-any-Origin behavior this
change exists to prevent. Introduce CORS_ALLOW_CREDENTIALS keyed on the
parsed list containing '*' and use it at both the middleware registration
and the manual error-response headers; replace the tautological formula
tests with ones exercising the real parser.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: don't leak filesystem info via Vertex credential test errors
Vertex's credentials_path is free text with no path validation, and
Google's auth library raises distinguishable exceptions for "file
missing" (OSError), "not valid JSON" (json.JSONDecodeError), and "valid
JSON but wrong shape" (GoogleAuthError). Echoing any of these back to an
API caller turns credential/model testing into a filesystem oracle - an
attacker who can create/test a Vertex credential could probe for the
existence and contents-shape of arbitrary files on the server.
_is_vertex_credentials_file_error() catches all three and both call
sites (api/credentials_service.py's test_credential, connection_tester's
test_individual_model) return one generic message instead.
* fix: don't classify network failures as credentials-file errors
ConnectionError/TimeoutError are OSError subclasses and TransportError is
a GoogleAuthError subclass, so the oracle guard also swallowed genuine
network failures - a user with a blocked network would be told to go debug
their credentials file. Excluding them reveals only the error's category,
which keeps the filesystem oracle closed.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: sanitize raw HTML in the note/transformation markdown preview
MarkdownEditor's live preview renders through @uiw/react-markdown-
preview, which parses literal HTML in the markdown source into real
elements (its `raw` default) - including a live <iframe>. Notes can
hold AI-generated content that echoes an indirect prompt injection
from an ingested document, so this was reachable without the user
writing any HTML themselves.
Add rehype-sanitize (default schema) ahead of rehype-katex in the
preview pipeline. Ordering matters: sanitizing after katex strips
katex's own generated markup (not in the default allowlist), while
sanitizing before it only touches the raw-HTML-derived tree and
leaves not-yet-rendered math nodes alone. Verified against the real
preview component that this strips <iframe>/<script>/<style>/
javascript: URLs while fully preserving math, syntax highlighting,
and GFM tables/task-lists.
* fix: make validate_url() async so DNS resolution doesn't block the event loop
socket.getaddrinfo() in validate_url()'s hostname-resolution branch ran
synchronously on every call - and this is called on the hot path of model
provisioning, potentially once per chat message/transformation. A slow or
hanging DNS lookup stalled every other concurrent request, not just the
one that triggered it.
Run the resolution via asyncio.to_thread() instead, and thread `await`
through every call site: credentials_service.py's discover functions,
routers/credentials.py's create/update, routers/sources.py's source-URL
ingestion, and connection_tester.py's three provider test functions.
save_uploaded_file() did a plain synchronous open()/write() directly
in the async create_source handler, blocking the event loop - and
every other concurrent request - for the duration of a large upload.
Same bug class the recent chat-graph fix (#971) addressed, just not
applied here.
Move the filesystem work (filename resolution + write) into a sync
helper run via asyncio.to_thread(), matching the pattern already used
for execute_command_sync elsewhere in this file. Confirmed the event
loop stays responsive (ticking normally) during a simulated slow
write, and that errors/cleanup still propagate correctly through the
thread.
Building default chat context, the notebook context endpoint, and
podcast generation all looped over every source in a notebook calling
get_context() -> get_insights(), each a separate query that also pays
its own connection setup (no pooling). A notebook with hundreds of
sources meant hundreds of serialized round trips before a chat
message even reached the LLM.
Add SourceInsight.get_for_sources() to fetch insights for every
source in one query, and thread an optional pre-fetched insights list
through Source.get_context() so callers can opt in without changing
its behavior for anyone who doesn't. Measured against a real
(embedded) SurrealDB instance: 14 queries down to 3 for a 12-source
notebook, with correctness verified. The two router call sites treat
a batch-fetch failure the same way the old per-source loop treated a
single failure - falls back to empty insights rather than failing the
whole request.
validate_url() only ran when a credential was created or updated. The
actual HTTP requests (connection testing, model discovery, and real
inference through Esperanto) re-resolve DNS fresh on every call, so a
hostname that resolved to a public IP at save time can later be
repointed to an internal or cloud-metadata address - a classic
DNS-rebinding TOCTOU that a one-time check can't catch.
Move validate_url() to open_notebook/utils so it can be re-run from
the AI layer without an api-depends-on-open_notebook layering
violation, and re-check immediately before every outbound request:
connection_tester's three providers, credential model discovery, and
ModelManager.get_model() on the real-inference path. Also close a
second gap in the same validator: it missed AWS's IPv6 metadata
address (fd00:ec2::254), which isn't link-local so the existing check
never caught it.
POST /sources with type=link copied the user-supplied URL straight
into content-core's fetch with zero validation - unlike the
credential-URL path, which already blocks internal/metadata
addresses. Any user could make the server fetch cloud metadata
endpoints or scan the internal network via "add a web source".
Reuse the same validate_url() guard at the point the URL is first
accepted, before it's ever handed to content-core.
