* fix(sources): fall back to auto when a selected engine's runtime is absent
The content-processing engine choice is persisted in the database; the
runtime that serves it (Docling, local Crawl4AI) is installed on demand
from environment flags evaluated at boot. The two therefore drift: a
redeploy that drops OPEN_NOTEBOOK_ENABLE_CRAWL4AI/_DOCLING, a volume
moved to a new deployment, or a failed on-demand install all leave a
stored selection pointing at a runtime that is not there.
The source graph passed that selection straight to content-core, so
every affected extraction failed with "Could not extract any text
content from this source" - no mention of the engine, the runtime, or
the flag that would fix it. For a URL engine set to crawl4ai this breaks
URL ingestion entirely.
The graph now checks runtime availability before honoring the stored
engine and degrades to content-core's "auto" chain, logging a WARNING
that names the engine and the env var that would enable it. Engines with
no opt-in runtime (auto/simple/firecrawl/jina) are passed through
untouched.
The availability probes moved from api/routers/capabilities.py to
open_notebook/utils/runtime_capabilities.py so the graph can use them
without importing from the API layer; the capabilities endpoint keeps
identical behavior and its tests follow the probes to their new home.
Found by the smoke-e2e agent during v1.14.0 release testing, on a dev
environment that was in exactly this state. Pre-existing since v1.13.0
(#1122 made the runtimes opt-in, #432 made the stored selection take
effect), not a v1.14.0 regression.
* docs(changelog): record the unavailable-engine fallback fix
Surface content-core 2.x's docling_formulas (formula extraction) and
docling_vision (image/chart vision) enrichment flags in Settings →
Content Processing, mirroring the existing OCR toggle. Both default off
and are gated on Docling availability in the UI. The settings persist
via GET/PUT /api/settings and are threaded into content-core extraction
alongside docling_ocr. Migration 23 backfills the new fields on the
existing content_settings record. Labels and help are translated across
all 14 locales.
Closes#1131
* fix(models): stop auto-assign from re-filling cleared optional defaults
Auto-assign treated every empty default slot as "missing" and filled it,
so an optional slot a user deliberately cleared (to fall back to the chat
model) got silently re-populated on the next run, undoing the intent.
- Auto-assign now fills only the required slots (chat, embedding); the
optional slots (transformation, tools, large context, TTS, STT) are
left untouched.
- get_default_model("large_context") now falls back to the chat model
when unset, matching transformation/tools (TTS/STT still return None).
- Settings UI shows an inline hint on each empty optional slot: the text
slots show "using chat model (<name>)"; TTS/STT show a not-configured
hint. Required slots remain non-clearable. New i18n keys across all 14
locales.
Closes#1098
* test: guard await_args against None for mypy
Register anthropic_compatible in the provider registry so its env config,
modalities, test model, and /api/providers entry are derived from PROVIDERS
(#1075's single source of truth); the only manual copy is the SupportedProvider
Literal. Maps to esperanto's anthropic provider with a custom base_url, and
re-injects that base_url via ChatAnthropic since esperanto's to_langchain drops
it. Connection-test and model discovery mirror the openai_compatible siblings,
including DNS-rebinding pinning (prepare_pinned_http_target). A single shared
validator enforces the base_url + api_key requirement on both the create and
update paths.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Expose Esperanto's built-in omlx OpenAICompatibleProfile in Settings with
Ollama-style UX (default http://localhost:11435/v1, optional API key,
language+embedding discovery). No openai-compatible remapping or
OPENAI_COMPATIBLE_* env mirroring.
Fixes#1048
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Wire four esperanto 2.25.1 providers into the provider matrix:
- Cohere (COHERE_API_KEY): language + embedding via the native v2 API,
with a bespoke discoverer (AIFactory.get_provider_models) since Cohere
is not OpenAI-compatible. Reranking is out of scope (#1087).
- Deepgram: extend modalities to add speech_to_text (Nova/Whisper)
alongside the existing Aura text_to_speech voices.
- PayPerQ / PPQ (PPQ_API_KEY): multi-modality OpenAI-compatible gateway
(https://api.ppq.ai/v1), auto-discovered via the /models endpoint.
- Novita (NOVITA_API_KEY): OpenAI-compatible LLM gateway
(https://api.novita.ai/openai), auto-discovered via /models.
Registry-derived surfaces (env config, modalities, test models, discovery
table, GET /api/providers) update automatically; the SupportedProvider
Literal, key_provider config, availability env map, docs and tests were
updated to match. Closes#1170
* fix(notebook): cascade-delete chat sessions on notebook deletion
Deleting a notebook removed its notes and exclusive sources but left
chat_session records orphaned. Extend Notebook.delete() to enumerate the
notebook's chat sessions via the existing refers_to relation and delete
each one, and report chat_session_count in the delete preview.
Closes#1124
* refactor(notebook): drop chat-session count from delete preview
The delete-preview API advertised chat_session_count but the frontend
dialog and locales never render it, creating a UI inconsistency. Trim
the preview back to notes and sources; the deletion cascade and its
post-delete deleted_chat_sessions count are unchanged.
* fix(sources): cap error text surfaced to clients
Source processing status (get_source_status) and sync-processing failures
returned the raw command/result error_message to clients unbounded, which
could leak arbitrary internal exception text and didn't match the error
capping applied elsewhere in the API.
Add a None-safe _truncate_error helper (200-char cap, ellipsis when cut)
and apply it on both paths. Adds focused unit tests for the helper.
Closes#1136
* test: narrow Optional return before assertions to satisfy mypy
Docling and local Crawl4AI are now opt-in, installed on first container startup
via OPEN_NOTEBOOK_ENABLE_DOCLING / OPEN_NOTEBOOK_ENABLE_CRAWL4AI, keeping the
default image lean (Crawl4AI un-bundled). Downloads cache on the /app/data
volume; installs are blocking with loud logs and degrade-don't-die on failure.
A GET /api/capabilities probe reports actual availability and the Settings UI
gates the Docling/Crawl4AI engines and OCR toggle (with env-var hints) until the
runtime is present. Docs folded from #1121 and rewritten to opt-in; ADR-007 added.
Closes#1122. Closes#1105.
Expose content-core's docling_ocr flag as a user setting. OCR runs on
scanned PDFs and images when the Docling engine handles them; it's on by
default (matching content-core) and can be turned off for faster
processing of text-native documents.
- New `docling_ocr` boolean on ContentSettings (default True), plumbed
through the settings API and into ContentCoreConfig in the source graph.
- "Enable OCR" checkbox in the Content Processing settings card, with
label + help translated across all 14 locales.
Part of #939. Stacked on #432.
* feat(sources): add Crawl4AI URL engine and honor persisted engine settings (#432)
- Add "crawl4ai" as a selectable URL processing engine (domain Literal,
settings API validation, SettingsForm select, and label across all 14
locales; urlHelp updated in en-US to describe the new fallback chain).
- The source graph now loads the persisted ContentSettings and passes the
document/URL engine choices to ContentCoreConfig. Previously it built a
hard-coded ContentSettings with "auto" engines, so a user's selection in
Settings never took effect. Falls back to defaults if settings can't load.
- Crawl4AI Docker mode is driven by content-core's native CRAWL4AI_API_URL
env var (documented separately under #1105).
Part of #939.
* fix(432): bundle Crawl4AI runtime + address review
- Bundle the Crawl4AI runtime so its local, no-API-key mode works out of
the box: depend on content-core[crawl4ai] and install the Chromium
browser via playwright in the Docker runtime-base (both image variants).
Footprint is modest (no torch/transformers/CUDA); image grows ~300 MB
from Chromium + system libs.
- Preserve the server-side traceback when persisted content settings fail
to load (logger.opt(exception=True)) instead of only the message.
- Reset the ContentSettings singleton between domain tests (clear_instance)
so a non-default value can't leak into neighboring tests.
Addresses review on #432.
* i18n(432): translate urlHelp Crawl4AI description across all 13 non-en locales
The Crawl4AI engine label was already localized; this brings the URL-engine
'help me choose' text in line with en-US in every locale — describing
Crawl4AI (local JS rendering, no API key) and its place in the auto
fallback chain (Firecrawl -> Jina -> Crawl4AI -> simple).
Unsupported files used to enqueue a background job that failed and then
burned the full 15-attempt retry budget (~1h) before showing a generic
"Failed" with no actionable detail.
- Add a pre-flight `_assert_file_supported()` using content-core 2.x's
header-only `check_file_support()` (same routing as real extraction) in
the upload branch of `_build_content_state`, before any job is enqueued
and before the source record is created.
- Guard the source-retry endpoint the same way. Unexpected check errors
(e.g. file removed before a retry) fall through to normal extraction.
- Map `UnsupportedTypeException` to `415 Unsupported Media Type` via a
dedicated global handler (previously fell through to the base
OpenNotebookError handler's 500). The message names the detected type.
Part of #939.
Since #1112 dropped the legacy provider/model strings, EpisodeCard's outline,
transcript and speaker rows showed "— / —" for new episodes. The API now
resolves the snapshot's model references (outline_llm/transcript_llm/voice_model)
to provider/name display fields at serialization time, batched into a single
query per request via Model.get_display_info_for_ids so listing episodes never
does a per-row model lookup. The card falls back to the legacy snapshot strings
for old episodes and degrades to a dash when a referenced model was deleted.
Closes#1114
Migration 22 best-effort maps profiles whose outline_llm/transcript_llm/
voice_model references are still empty to existing model records
(provider + name + type, no auto-create), clears the legacy values and
drops the 6 columns. The startup data migration that retried this
mapping on every boot (open_notebook/podcasts/migration.py) is deleted
along with its api/main.py lifespan hook; the legacy fields are removed
from the Pydantic models, API schemas and frontend types/panels.
Accepted trade-off: profiles whose mapping never converged lose the
legacy strings and stay unresolved - they were already non-functional
and the UI already flags them; the user re-picks models once.
* refactor(podcasts): store audio paths relative to PODCASTS_FOLDER
audio_file previously stored the absolute path returned by
podcast-creator (sometimes as a file:// URI), which kept path traversal
representable in the DB and broke playback when DATA_FOLDER moved.
- New single choke point open_notebook/podcasts/audio_paths.py:
to_relative_audio_path() validates at write time (the DB can never
hold an absolute or root-escaping value) and
resolve_contained_audio_path() joins + resolves + contains at read
time, replacing the per-router guards from #1018. Absolute/file://
legacy values are treated as invalid (same 403/404 as before).
- Generation command stores the relative form; in-band "ERROR: ..."
values from podcast-creator now fail the job with the real error.
- build_episode_output_dir() builds from PODCASTS_FOLDER so the write
root and the validation root cannot drift.
- Migration 21 converts legacy rows under the known roots (plain
file:/// URIs, /app/data/podcasts/, /data/podcasts/,
./data/podcasts/, data/podcasts/) via a single-strip IF/ELSE chain;
rows under other roots stay untouched by design.
Closes#1030
* test: use the tolerant migration-count assertion pattern
Per review: assert >= 21 with up/down parity and a distinctive-SQL check on
index 20, matching the convention from the migration-19/20 tests, so the next
migration doesn't trip an unrelated test file.
* fix(podcasts): reference speaker profiles by record ID instead of name
episode_profile.speaker_config now stores a record<speaker_profile>
reference (migration 20 converts existing rows; orphaned names become
null). The generate API keeps accepting the speaker profile by name and
resolves it to a record ID at the boundary.
Closes#630
* fix(podcasts): survive orphaned speaker references in podcast-creator config
Review fixes: rewrite record IDs back to speaker names (and drop orphaned
profiles) when building podcast-creator's episode config so one orphaned
profile can't fail validation for every generation; clearer error for
dangling references; resolve a single speaker name on single-record
endpoints instead of fetching the whole speaker table; accurate wording
about record links not preventing dangling references.
* fix(podcasts): submit the speaker profile record ID from the generate dialog
Submitting the cached display name made generation fail if the speaker
profile was renamed while the dialog was open - the record ID is stable
and the API resolves either form.
source_insight was the only content table without created/updated field
definitions. Since the table is SCHEMAFULL and insights are created via a
raw CREATE (create_insight command), SurrealDB silently dropped any
timestamps - rows genuinely held NONE - and the API then wrapped them in
str(), so clients received the literal string "None".
- Migration 19 defines created/updated on source_insight with the same
time::now() defaults used by source, note and notebook, so new insights
are stamped at creation. Existing rows are left untouched (no backfill).
- SourceInsightResponse.created/updated are now Optional[str]; both
routers emit an ISO 8601 string when the timestamp is present and null
when it is absent (legacy rows), never "None".
- Frontend types updated to string | null accordingly.
- New tests cover the migration definition/registration and the API
serialization (absent -> null, present -> ISO string).
Note: Python-side stamping alone was not viable - the table is SCHEMAFULL,
so undefined fields written by the client are silently dropped on
SurrealDB 2.x (verified against a live instance), hence the schema
migration mirroring the other tables.
Closes#1045
PUT /api/models/defaults used 'is not None' guards, so an explicit null
sent to clear a default was silently ignored — the old value survived
while the client saw success (same anti-pattern fixed for credentials
in #1046). The handler is now keyed on field presence (model_fields_set):
absent keeps the current value, explicit null clears it. The required
defaults (chat, embedding) reject null with a 400.
In the UI, the optional default-model selects (transformation, tools,
large context, TTS, STT) now offer a clear option — labeled 'Use
fallback (chat default)' for transformation/tools, which fall back to
the chat default when unset, and 'None' otherwise — and display that
state when no model is assigned. The transformation default is no
longer marked required, matching the backend fallback behavior.
Closes#1091
Consolidate the three copies of context assembly into
open_notebook/utils/context_builder.py:
- POST /api/chat/context now delegates to build_notebook_context()
(same request/response shapes, same string-matching config semantics)
- The source-chat graph now calls build_source_context() instead of the
495-line generalized ContextBuilder class, which had exactly one
caller and whose notebook/notes/priority-config flexibility was dead
- POST /api/notebooks/{notebook_id}/context removed: it duplicated
/api/chat/context with a slightly different envelope and had zero
callers (frontend, docs, tests)
Behavior is pinned by new characterization tests written before the
refactor (tests/test_context_endpoint_characterization.py) plus unit
tests for build_source_context.
The routers wrapped endpoint bodies in a broad 'except Exception' that
re-raised everything as HTTPException(500), intercepting the typed
open_notebook.exceptions hierarchy before the global handlers in
api/main.py could map it to its documented status code (NotFoundError
-> 404, InvalidInputError -> 400, ConfigurationError -> 422,
RateLimitError -> 429, NetworkError/ExternalServiceError -> 502).
Every endpoint-level handler chain across the 18 affected routers now
re-raises HTTPException and OpenNotebookError before the final generic
arm, which keeps catching untyped exceptions and returning a sanitized
500 (the no-raw-exception-text guarantee is unchanged and covered by
tests/test_error_message_sanitization.py). Nested best-effort blocks
(fallbacks, SSE error events, per-item loops) are deliberately left
untouched. In create_source and _create_source_async_path the new
re-raise arms preserve the uploaded-file / orphaned-source cleanup.
Characterization tests that pinned the old wrapped-500 behavior for
source-chat sessions with a missing refers_to relation are updated to
the correct 404, and a new test module asserts per router that a
domain-layer ConfigurationError maps to 422 instead of 500.
* ci: gate PRs on mypy, start ignore_errors burn-down
Add a backend-typecheck CI job running uv run python -m mypy . and bring
the repo-wide baseline from 197 errors to 0 so the gate blocks new type
errors from now on:
- enable the pydantic mypy plugin (resolves 138 false positives on
models whose fields have Field(None, ...) defaults)
- fix the remaining errors with real annotations; the only new type:
ignore comments cover a genuine langgraph typing limitation (partial
state dicts are valid at runtime but the overloads require the full
state type) and tests that intentionally pass invalid input
- start the ignore_errors burn-down: open_notebook.graphs.transformation,
open_notebook.graphs.ask and api.routers.models are now type-checked;
stale blocks for the deleted api.client and api.podcast_api_service
modules removed. Only open_notebook.domain.notebook remains exempt
(DB layer is migrating to surreal-basics)
- fix the mypy.ini header comment to describe what the config does
* test: use typing.get_args on the registry literal check
The registry test landed in parallel using Literal.__args__, which the
mypy gate in this branch rejects; align it with the get_args() idiom
used by the rest of the file.
* refactor(ai): single provider registry as the backend source of truth
Provider metadata (env vars, modalities, connection-test models,
OpenAI-compatible discovery URLs, display names, docs links) is now
defined once in open_notebook/ai/provider_registry.py. The existing
surfaces are derived from it, keeping every import and call-site shape
unchanged:
- api/credentials_service.py: PROVIDER_ENV_CONFIG, PROVIDER_MODALITIES
and the discovery url_map are built from the registry
- open_notebook/ai/connection_tester.py: TEST_MODELS derived
- open_notebook/ai/model_discovery.py: OPENAI_COMPAT_PROVIDERS built
from registry entries with a discovery URL (quirk hooks stay local)
The SupportedProvider Literal (typing, can't be built at runtime) and
the frontend provider tables remain manual copies; the cross-check
tests now assert registry keys == Literal == frontend list, plus
registry internal consistency and discovery-table coverage.
New GET /api/providers endpoint exposes the registry (name, display
name, modalities, docs_url, env-configured status) so clients can stop
hardcoding provider lists (frontend adoption is a follow-up).
Docs updated: open_notebook/AGENTS.md and docs/7-DEVELOPMENT/credentials.md
now describe the registry instead of the four-place sync rule.
* refactor(ai): address review findings on the provider registry
- Build PROVIDERS via _build_registry(), which raises on a duplicate
provider name at import time instead of silently dropping the earlier
spec (dict-comprehension behavior); regression test added
- Pin the exact OpenAI-compatible provider -> discovery URL mapping in
a test so a registry edit can't silently drop or misassign a URL
- Give TEST_MODELS a real type annotation
(Dict[str, Tuple[Optional[str], str]]) instead of bare dict
- Extract _source_to_response() shared builder (SourceResponse was
hand-rolled 5x with the same nested AssetModel ternary)
- Extract _cleanup_uploaded_file() (the unlink ritual was pasted 6x
inside create_source); inner cleanups were redundant with the outer
exception handlers, which always run for the same failures
- Split create_source into _build_content_state() (type validation +
SSRF/LFI guards) and _create_source_async_path() /
_create_source_sync_path()
- Unify the paginated list query (with/without notebook filter differed
only in the FROM clause and bound params)
Pure structural refactor: same status codes, error messages and
response shapes. Security checks (atomic filename claim via
touch(exist_ok=False), path-traversal containment, SSRF/LFI guards)
preserved verbatim, with their comments.
* test(api): characterize shared chat/source-chat router behaviors
* refactor(api): extract shared session and message helpers for chat routers
* refactor(api): mark intentionally unused source id unpacks
The session handlers only need the verified session; the source-level
verification happens inside get_verified_source_session. Underscore the
unused binding in get/update/delete to make that explicit.
Remove F401 (unused imports), F841 (unused local variables) and E722
(bare except) from the ruff ignore list; the legacy Streamlit-era code
that motivated ignoring them is gone.
Fallout fixed: 10 unused imports removed (none were load-bearing
re-exports or side-effect imports), 2 unused mock bindings in tests
dropped, and a now-empty TYPE_CHECKING block in
open_notebook/utils/embedding.py cleaned up. No bare excepts remained
in the codebase.
Delete api/client.py (synchronous httpx client calling the app's own
FastAPI server, a leftover from the removed Streamlit UI) and the 13
api/*_service.py wrappers around it. None of these files had any
importer in routers, commands or tests, verified by grepping the whole
repo for each module name.
Also remove commands/example_commands.py (process_text/analyze_data
demo commands from the surreal-commands README) and its export from
commands/__init__.py, and update remaining docstring examples to
reference the real generate_podcast command.
The real services (command_service, credentials_service,
podcast_service) are untouched.
* fix(frontend): actually clear credential fields when emptied in the edit dialog
Clearing base_url (or the Vertex project/location/credentials_path)
in the credential edit dialog silently did nothing: the submit handler
mapped an emptied field to `undefined`, JSON.stringify dropped the key
from the PUT body, and the backend's partial-update semantics kept the
old value — while the UI reported success.
Emptied fields are now sent as an explicit `null`, which the API
already accepts and persists as a cleared value (verified live).
UpdateCredentialRequest's nullable fields are typed accordingly.
Found in v1.11 release testing (pre-existing, not a release regression):
an Ollama credential with a stale IP in base_url could not be cleared
from the UI.
* fix(api): clear credential fields on explicit null, not just empty string
The update handler guarded every field with 'is not None', so a JSON
null sent to clear base_url (or endpoint/api_version/endpoint_*/
project/location/credentials_path/num_ctx) was silently skipped — the
old value survived while the client saw a 200. Combined with the
frontend bug fixed in the previous commit (emptied fields dropped from
the payload entirely), clearing a credential field was impossible from
the UI: during release testing an Ollama credential kept pointing at a
stale IP after being 'cleared', and only an empty-string PUT crafted by
hand actually cleared it.
Field updates are now keyed on presence in model_fields_set: absent
keeps the old value, explicit null or "" clears. name/modalities/
api_key keep their non-null guards (clearing those is not meaningful).
Regression tests cover null-clears, empty-string-clears, absent-keeps
and the Vertex credentials_path case.
* test(frontend): extract credential update payload builder and cover clear-on-empty
Addresses the cubic review: the frontend half of the fix had no
reproducing test. The edit dialog's payload construction now lives in
a pure buildCredentialUpdatePayload() (frontend/src/lib/
credential-update-payload.ts) used by the component, with tests that
fail on the original bug: an emptied base_url (and Vertex project/
location/credentials_path) must survive JSON.stringify as an explicit
null, unchanged fields must be omitted, and emptied num_ctx clears
via 0.
Two findings from v1.11 release testing:
- GET /api/sources?sort_by=title returned a 500. source.title carries a
SEARCH (BM25) index (idx_source_title, migration 1) and SurrealDB's
planner fails ORDER BY on such a column with 'No iterator has been
found'. The query now sorts by a computed alias
(string::lowercase(title OR '') AS title_sort), which sidesteps the
index and makes the sort case-insensitive as a bonus.
- POST /api/sources with an over-limit notebooks/transformations array
(or invalid JSON in either field) returned a raw 500.
parse_source_form_data() builds SourceCreate manually, so pydantic's
ValidationError never reached FastAPI's request-validation handler.
Both cases now surface as a clean 422 with a descriptive message.
Verified against a live SurrealDB v2 instance; regression tests added
for the ORDER BY alias, all six sort fields, and the 422 paths.
* fix: update Google model version in connection tester and tests
* fix: updated the gemini model lists that references deprecated models
* fix: update also documentation. This is more prone to not follow the maintainer's directives, so PTADL
* fix: forgot two references to gemini deprecated versions
* fix: use valid, longer-lived gemini model IDs
Several IDs the PR introduced don't exist or are near shutdown, verified
against Google's official model/deprecation pages (2026-07):
- connection test model gemini-2.5-flash -> gemini-3.5-flash (2.5-flash
retires 2026-10-16; 3.5-flash is the current stable GA and Google's
named replacement, so the #970 fix doesn't re-break in ~3 months)
- gemini-3.5-pro (does not exist) -> gemini-2.5-pro in docs
- plain gemini-3.1-flash (not a GA Gemini-API id) dropped from the Vertex
discovery list / preferences / docs; use gemini-3.5-flash or -flash-lite
- dead gemini-pro dropped from preferences and docs
Ported #996's #970 regression test, pinned to gemini-3.5-flash.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: surface silent command-submission failures where they matter
Source.add_insight() caught submission failures and returned None
instead of raising - callers (transformation.py, source.py) run inside
surreal-commands jobs whose outer exception handling already
retries/fails on this, so a swallowed submission failure meant a
transformation could report success while the insight was silently
never persisted. Now raises DatabaseOperationError, matching
vectorize()'s existing contract.
Note.save()'s auto-embed needs the opposite treatment: it's an implicit
side effect of save() (not an explicit dedicated call), and the note
itself is already durably saved by the time it runs - so a submission
hiccup there shouldn't turn an otherwise-successful save into a 500.
Wrapped in try/except, logs and returns None. api/routers/embedding.py's
explicit POST /embed (item_type=note) is the one caller for whom
submission success genuinely is the point of the call, so it separately
checks for a missing command_id and surfaces that as a failure.
* fix: collapse duplicate error logging in add_insight
logger.error + logger.exception logged the same failure twice; a single
logger.exception call carries both the message and the traceback (same
form vectorize() already uses).
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Two small fixes to the podcast episode-listing path, plus a doc note:
- audio_file is only ever set server-side today from a UUID-named
directory under PODCASTS_FOLDER, so this can't currently be tripped -
but the stream/retry/delete endpoints didn't verify the resolved path
actually stayed within PODCASTS_FOLDER before following it. Add
_is_audio_path_contained() as defense in depth against a future code
path (e.g. importing external audio) setting audio_file to something
else.
- Listing episodes called get_job_detail() -> get_command_status() once
per episode, each its own round trip (no connection pooling). Add
PodcastEpisode.get_job_details_for_commands() to batch-fetch status
for every episode's command in one query instead.
Also documents (docs/7-DEVELOPMENT/security.md) that podcast_creator's
configure("templates", {...}) compiles strings as Jinja2 template
source - the same shape as the SSTI vulnerability fixed in
transformation.py (GHSA-f35w-wx37-26q7). Confirmed dormant: no code path
in this repo calls it today. commands/podcast_commands.py gets a
matching code comment warning against wiring user text into it if a
"custom podcast template" feature is ever added.
api/routers/sources.py and api/podcast_service.py interpolated the raw
exception (detail=f"...: {str(e)}") into client-facing error responses -
inconsistent with the safer pattern already used elsewhere in the same
files (e.g. the download handlers), which log the raw exception
server-side but return a fixed generic message. Internal details (DB
hostnames, connection errors, stack-trace fragments) could leak to any
API caller through a 500 response.
Every occurrence already had a matching logger.error() call, so this is
a client-facing message change only, not a logging change. Deliberate
app-authored messages (InvalidInputError text, "Notebook X not found",
result.error_message) are untouched - only raw f"{str(e)}" interpolation
is affected.
tests/test_config_endpoint_no_leak.py is a regression lock for
api/routers/config.py (already correct, not touched by this diff) rather
than a fix - added since it shares the same "unauthenticated endpoint,
don't leak exception text" concern.
Three small, independent hardening fixes to source ingestion:
- generate_unique_filename() checked `if not resolved.exists()` then let
a separate write happen later - two concurrent uploads landing on the
same candidate name could both pass the check and clobber each other.
Now atomically claims the name via Path.touch(exist_ok=False) (O_EXCL)
as part of the search loop itself.
- _resolve_source_file() and _is_source_file_available() compared
`resolved_path.startswith(safe_root)` without a trailing separator - a
sibling directory that merely starts with the same string (e.g.
"uploads_evil/") would incorrectly be treated as contained, unlike this
file's other two path checks which already guard with `+ os.sep`. Not
reachable today (source.asset.file_path is only ever set server-side),
but this closes the gap and matches the existing pattern.
- SourceCreate.notebooks/transformations had no length limit; both are
iterated with a per-item DB lookup in create_source(), so an unbounded
array let a single request trigger an unbounded number of sequential DB
round trips. Capped at 50.
tests/test_upload_type_mitigations.py adds no code change - it documents
why the adjacent "no file type allowlist on uploads" finding was
investigated and judged low-risk without one (downloads are already
served as application/octet-stream regardless of actual file type).
* fix: sanitize raw HTML in the note/transformation markdown preview
MarkdownEditor's live preview renders through @uiw/react-markdown-
preview, which parses literal HTML in the markdown source into real
elements (its `raw` default) - including a live <iframe>. Notes can
hold AI-generated content that echoes an indirect prompt injection
from an ingested document, so this was reachable without the user
writing any HTML themselves.
Add rehype-sanitize (default schema) ahead of rehype-katex in the
preview pipeline. Ordering matters: sanitizing after katex strips
katex's own generated markup (not in the default allowlist), while
sanitizing before it only touches the raw-HTML-derived tree and
leaves not-yet-rendered math nodes alone. Verified against the real
preview component that this strips <iframe>/<script>/<style>/
javascript: URLs while fully preserving math, syntax highlighting,
and GFM tables/task-lists.
* fix: make validate_url() async so DNS resolution doesn't block the event loop
socket.getaddrinfo() in validate_url()'s hostname-resolution branch ran
synchronously on every call - and this is called on the hot path of model
provisioning, potentially once per chat message/transformation. A slow or
hanging DNS lookup stalled every other concurrent request, not just the
one that triggered it.
Run the resolution via asyncio.to_thread() instead, and thread `await`
through every call site: credentials_service.py's discover functions,
routers/credentials.py's create/update, routers/sources.py's source-URL
ingestion, and connection_tester.py's three provider test functions.
save_uploaded_file() did a plain synchronous open()/write() directly
in the async create_source handler, blocking the event loop - and
every other concurrent request - for the duration of a large upload.
Same bug class the recent chat-graph fix (#971) addressed, just not
applied here.
Move the filesystem work (filename resolution + write) into a sync
helper run via asyncio.to_thread(), matching the pattern already used
for execute_command_sync elsewhere in this file. Confirmed the event
loop stays responsive (ticking normally) during a simulated slow
write, and that errors/cleanup still propagate correctly through the
thread.
Building default chat context, the notebook context endpoint, and
podcast generation all looped over every source in a notebook calling
get_context() -> get_insights(), each a separate query that also pays
its own connection setup (no pooling). A notebook with hundreds of
sources meant hundreds of serialized round trips before a chat
message even reached the LLM.
Add SourceInsight.get_for_sources() to fetch insights for every
source in one query, and thread an optional pre-fetched insights list
through Source.get_context() so callers can opt in without changing
its behavior for anyone who doesn't. Measured against a real
(embedded) SurrealDB instance: 14 queries down to 3 for a 12-source
notebook, with correctness verified. The two router call sites treat
a batch-fetch failure the same way the old per-source loop treated a
single failure - falls back to empty insights rather than failing the
whole request.
POST /sources with type=link copied the user-supplied URL straight
into content-core's fetch with zero validation - unlike the
credential-URL path, which already blocks internal/metadata
addresses. Any user could make the server fetch cloud metadata
endpoints or scan the internal network via "add a web source".
Reuse the same validate_url() guard at the point the URL is first
accepted, before it's ever handed to content-core.
repo_relate() interpolated the relate target directly into the query
string, reachable via an unvalidated notebook_id on the save-insight-
as-note flow - a crafted ID could inject and execute arbitrary
SurrealQL (confirmed against a live embedded instance: a single
crafted RELATE call wiped an entire table). Bind record identifiers
as query parameters instead of building them into the query text, and
validate notebook_id exists before relating, matching the pattern
already used by every other caller of add_to_notebook().
* feat: Recently Viewed section for notebooks and sources
Fixes#850
* fix: export RecentlyViewedResponse type from api types
The RecentlyViewed component and notebooks API client import
RecentlyViewedResponse from @/lib/types/api but the interface was missing,
failing the type-check build. Add it to match the backend api/models.py model.
* fix: harden recently-viewed read path and index the recency query
Three robustness fixes on the recently-viewed feature:
- Make the last_viewed_at write-on-read stamping best-effort. _stamp_source_view
and _stamp_notebook_view now swallow and log their own errors, so a failed
stamp update can no longer turn a successful GET /sources/{id} or
GET /notebooks/{id} into a 500.
- Stop leaking internal details from GET /recently-viewed: log the full
exception server-side and return a generic error message to the client.
- Add indexes on last_viewed_at for the notebook and source tables in
migration 16 so the ORDER BY last_viewed_at DESC LIMIT recently-viewed
query does not degrade into a full table scan as data grows. The down
migration removes the indexes before dropping the field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: register migration 18, add recently-viewed i18n keys and CHANGELOG entry
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* feat: per-transformation custom model selection
Fixes#776
* fix: validate model_id exists when creating or updating a transformation
The create and update endpoints persisted model_id without checking the
referenced model exists, so an invalid reference was stored silently and only
surfaced later as a 404 at execution time. Add the same existence check
execute_transformation already performs (Model.get -> 404 if missing) to both
the create and update paths, so a bad model_id is rejected up front with a
clear 404. Update still allows clearing model_id to None.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: cover model_id existence validation on transformation create/update
The create test now mocks Model.get so the new existence check passes and
asserts it was awaited with the supplied model_id. The update test expects
Model.get to be awaited twice (once validating the update, once at execute)
rather than once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for per-transformation model selection (#776)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: pass max_tokens through to podcast_creator for outline/transcript generation
* fix: persist and expose episode_profile max_tokens (migration + API)
* docs: add CHANGELOG entry for episode profile max_tokens pass-through (#639)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Expand /api/sources sorting to type, title, created, updated, insights_count, and embedded (whitelisted, with a computed type alias so SurrealDB can order by it). Frontend adds sortable headers for every column plus an Updated column.
Fixes#895
Notebook chat (execute_chat) and source-chat SSE ran LangGraph's synchronous invoke() on the event loop, freezing the whole API while the LLM responded. Both now run via asyncio.to_thread(), matching the existing get_state() calls.
Fixes#704
Switches the streaming endpoints (/api/search/ask, source chat messages) to media_type=text/event-stream and adds dedicated App Router SSE route handlers (_sse-proxy.ts) that stream the upstream body directly, bypassing the Next.js rewrites proxy's gzip buffering in standalone/production mode.
Verified locally: with Accept-Encoding: gzip (i.e. a real browser), the standalone rewrites proxy buffered the entire SSE stream to completion; the route-handler approach restores progressive delivery.
* fix(api): return 404 instead of 500 for missing resources in CRUD endpoints (#862)
ObjectModel.get() raises NotFoundError for a missing record (never returns a
falsy value), so the 'if not obj: 404' guards were dead code and each handler's
broad 'except Exception' re-raised NotFoundError as 500, never hitting the global
NotFoundError->404 handler.
Add an explicit 'except NotFoundError -> 404' arm to the affected handlers in
notebooks, notes, models, credentials and embedding routers, plus regression
tests asserting 404 (not 500) when .get() raises NotFoundError.
* refactor(api): remove dead 'if not obj' guards after .get() (cubic #924)
Now that each handler maps NotFoundError -> 404, the 'if not obj: raise 404'
guards are unreachable (.get() raises rather than returning a falsy value).
Drop them; where the fetched object was only used by the guard, keep the .get()
call for its existence-validation side effect without binding an unused var.
* chore(release): prepare v1.10.0
* style: fix import ordering flagged by ruff
* fix(sources): return 404 for missing source and fix retry 500 from double-prefixed command id
- GET /sources/{id} mapped NotFoundError to a generic 500; now returns 404
- POST /sources/{id}/retry double-prefixed the command id (command:command:...),
raising 'too many values to unpack' after queuing; align with the create path
- update retry test mock to realistic prefixed command id + guard against
double-prefix; add 404 regression test
* fix(sources): mark failed extraction as failed so retry surfaces (#726)
content-core signals soft failures (unreachable/invalid URL) by returning
title=Error + 'Failed to extract content:' body instead of raising, and the
process_source command swallowed permanent ValueErrors into a success=False
result. Since surreal-commands marks a job completed when the function returns,
failed ingests showed status 'completed' and never offered the retry button.
- source graph: detect the content-core failure sentinel and raise
- process_source_command: re-raise ValueError (stop_on already prevents retry)
so the job is marked failed and the source becomes retryable
* feat(notebook): per-type bulk context actions for sources and notes (#223)
- Sources context menu now offers 'insights only' (sources without insights
are excluded rather than forced to full), 'full content', and 'exclude all'
- Add the same bulk Context menu to the Notes column (include all / exclude all)
- Bulk choices propagate to items loaded later via pagination
- New locale keys (includeAllInsights/includeAllFull) across all 14 locales
- Unit tests for the new bulk modes and note context helpers
POST /sources/{id}/retry looked up a source's notebooks with
`SELECT notebook FROM reference WHERE source = $source_id`, but `reference`
is a graph edge (RELATE source->reference->notebook) with only `in`/`out`
columns. The query matched nothing, so `notebook_ids` was always empty and
the endpoint returned 400 for every source. Mirror the working query used in
the source-list path: `SELECT VALUE out FROM reference WHERE in = $source_id`.
Adds regression tests asserting retry re-queues a linked source and only 400s
when a source is genuinely unlinked.
PUT /api/episode-profiles/{id} and PUT /api/speaker-profiles/{id}
both blindly assigned every field of the request body model (including
Optional fields with None defaults) onto the DB object. SurrealDB
rejected None values for fields typed as string — causing HTTP 500
whenever the client omitted any optional field.
Replace the per-field assignment blocks with a model_dump(exclude_unset=True)
loop so only fields the client actually included are written through.
Fields absent from the request body retain their current DB values.
Closes#809
Co-authored-by: james <li@jamesdeMacBook-Pro.local>
Surface the audio providers added in esperanto 2.21/2.22:
- Mistral Voxtral STT (voxtral-*-latest) and TTS (voxtral-mini-tts) — reuses
MISTRAL_API_KEY; discovered via the existing /v1/models endpoint.
- Deepgram TTS (Aura) — new provider: DEEPGRAM_API_KEY env config, key_provider
mapping, static Aura voice catalog for discovery, modality + display name +
docs link in the UI.
- xAI TTS — adds text_to_speech to the existing xai provider. xAI TTS is
voice-based and sends no model id, so the model name is cosmetic; users add it
via the custom-model input (no discovery entry).
classify_model_type now distinguishes Voxtral TTS vs STT (the "-tts" model must
not be caught by the broader STT names, since STT is checked first) and Aura
voices. PROVIDER_MODALITIES, PROVIDER_ENV_CONFIG, discover_with_config static
list, TEST_MODELS, DEFAULT_TEST_VOICES, the provider-availability endpoint, and
the frontend provider constants are all updated. Provider display names are
frontend constants (not i18n), so no locale changes are needed.
Also removes the dead test_provider_connection() function (no callers; the UI
uses test_credential/test_individual_model) and its now-unused imports.
Closes#826Closes#827
Upgrade esperanto 2.20.0 -> 2.22.0. The constraint (>=2.20.0,<3) already
allowed it; this relocks and picks up upstream fixes (OpenRouter json body,
clearer null-embedding errors, streaming ToolCall objects, base_url
trailing-slash normalization, Ollama thinking-model content).
Esperanto 2.21.0 lowered the Ollama num_ctx default from 128000 to 8192 to
avoid OOM on consumer GPUs. We keep that safe default and add an optional
per-credential num_ctx override for self-hosters whose hardware can handle a
larger context window:
- Credential gains a num_ctx field, surfaced via to_esperanto_config() so it
flows into AIFactory automatically (no ModelManager change needed).
- Credential create/update API schemas + router pass num_ctx through.
- Frontend: optional numeric field on the Ollama credential form, with i18n
labels translated across all 13 locales.
- Docs: document the new default and the override under AI providers.
Closes#825
