* feat(sources): add Crawl4AI URL engine and honor persisted engine settings (#432)
- Add "crawl4ai" as a selectable URL processing engine (domain Literal,
settings API validation, SettingsForm select, and label across all 14
locales; urlHelp updated in en-US to describe the new fallback chain).
- The source graph now loads the persisted ContentSettings and passes the
document/URL engine choices to ContentCoreConfig. Previously it built a
hard-coded ContentSettings with "auto" engines, so a user's selection in
Settings never took effect. Falls back to defaults if settings can't load.
- Crawl4AI Docker mode is driven by content-core's native CRAWL4AI_API_URL
env var (documented separately under #1105).
Part of #939.
* fix(432): bundle Crawl4AI runtime + address review
- Bundle the Crawl4AI runtime so its local, no-API-key mode works out of
the box: depend on content-core[crawl4ai] and install the Chromium
browser via playwright in the Docker runtime-base (both image variants).
Footprint is modest (no torch/transformers/CUDA); image grows ~300 MB
from Chromium + system libs.
- Preserve the server-side traceback when persisted content settings fail
to load (logger.opt(exception=True)) instead of only the message.
- Reset the ContentSettings singleton between domain tests (clear_instance)
so a non-default value can't leak into neighboring tests.
Addresses review on #432.
* i18n(432): translate urlHelp Crawl4AI description across all 13 non-en locales
The Crawl4AI engine label was already localized; this brings the URL-engine
'help me choose' text in line with en-US in every locale — describing
Crawl4AI (local JS rendering, no API key) and its place in the auto
fallback chain (Firecrawl -> Jina -> Crawl4AI -> simple).
Unsupported files used to enqueue a background job that failed and then
burned the full 15-attempt retry budget (~1h) before showing a generic
"Failed" with no actionable detail.
- Add a pre-flight `_assert_file_supported()` using content-core 2.x's
header-only `check_file_support()` (same routing as real extraction) in
the upload branch of `_build_content_state`, before any job is enqueued
and before the source record is created.
- Guard the source-retry endpoint the same way. Unexpected check errors
(e.g. file removed before a retry) fall through to normal extraction.
- Map `UnsupportedTypeException` to `415 Unsupported Media Type` via a
dedicated global handler (previously fell through to the base
OpenNotebookError handler's 500). The message names the detected type.
Part of #939.
Since #1112 dropped the legacy provider/model strings, EpisodeCard's outline,
transcript and speaker rows showed "— / —" for new episodes. The API now
resolves the snapshot's model references (outline_llm/transcript_llm/voice_model)
to provider/name display fields at serialization time, batched into a single
query per request via Model.get_display_info_for_ids so listing episodes never
does a per-row model lookup. The card falls back to the legacy snapshot strings
for old episodes and degrades to a dash when a referenced model was deleted.
Closes#1114
Migration 22 best-effort maps profiles whose outline_llm/transcript_llm/
voice_model references are still empty to existing model records
(provider + name + type, no auto-create), clears the legacy values and
drops the 6 columns. The startup data migration that retried this
mapping on every boot (open_notebook/podcasts/migration.py) is deleted
along with its api/main.py lifespan hook; the legacy fields are removed
from the Pydantic models, API schemas and frontend types/panels.
Accepted trade-off: profiles whose mapping never converged lose the
legacy strings and stay unresolved - they were already non-functional
and the UI already flags them; the user re-picks models once.
* refactor(podcasts): store audio paths relative to PODCASTS_FOLDER
audio_file previously stored the absolute path returned by
podcast-creator (sometimes as a file:// URI), which kept path traversal
representable in the DB and broke playback when DATA_FOLDER moved.
- New single choke point open_notebook/podcasts/audio_paths.py:
to_relative_audio_path() validates at write time (the DB can never
hold an absolute or root-escaping value) and
resolve_contained_audio_path() joins + resolves + contains at read
time, replacing the per-router guards from #1018. Absolute/file://
legacy values are treated as invalid (same 403/404 as before).
- Generation command stores the relative form; in-band "ERROR: ..."
values from podcast-creator now fail the job with the real error.
- build_episode_output_dir() builds from PODCASTS_FOLDER so the write
root and the validation root cannot drift.
- Migration 21 converts legacy rows under the known roots (plain
file:/// URIs, /app/data/podcasts/, /data/podcasts/,
./data/podcasts/, data/podcasts/) via a single-strip IF/ELSE chain;
rows under other roots stay untouched by design.
Closes#1030
* test: use the tolerant migration-count assertion pattern
Per review: assert >= 21 with up/down parity and a distinctive-SQL check on
index 20, matching the convention from the migration-19/20 tests, so the next
migration doesn't trip an unrelated test file.
* fix(podcasts): reference speaker profiles by record ID instead of name
episode_profile.speaker_config now stores a record<speaker_profile>
reference (migration 20 converts existing rows; orphaned names become
null). The generate API keeps accepting the speaker profile by name and
resolves it to a record ID at the boundary.
Closes#630
* fix(podcasts): survive orphaned speaker references in podcast-creator config
Review fixes: rewrite record IDs back to speaker names (and drop orphaned
profiles) when building podcast-creator's episode config so one orphaned
profile can't fail validation for every generation; clearer error for
dangling references; resolve a single speaker name on single-record
endpoints instead of fetching the whole speaker table; accurate wording
about record links not preventing dangling references.
* fix(podcasts): submit the speaker profile record ID from the generate dialog
Submitting the cached display name made generation fail if the speaker
profile was renamed while the dialog was open - the record ID is stable
and the API resolves either form.
The settings frontend now fetches the provider list from the backend
registry endpoint (session-cached react-query hook useProviders())
instead of keeping its own hardcoded copies of provider names, display
names, modalities and docs URLs in lib/providers.tsx.
- New api module (lib/api/providers.ts) + hook (lib/hooks/use-providers.ts)
with staleTime: Infinity — the list only changes on deploy.
- lib/providers.tsx reduced to modality presentation (icon/color/label)
behind fallback-safe helpers, so an unknown modality from a future
provider still renders instead of breaking.
- The backend registry declaration order is the display order (verified
identical to the old curated ALL_PROVIDERS order; endpoint test now
pins order, not just set-equality).
- api-keys page gains loading/error states for the provider fetch; new
i18n keys added to all 14 locales.
- Deleted the regex-based frontend/backend sync test; the
SupportedProvider Literal test remains the backend guarantee.
- Renamed the dead useProviders() in use-models.ts (availability
endpoint) to useProviderAvailability() to avoid a name collision.
- Updated stale docs/comments (AGENTS.md, credentials.md, cubic.yaml,
provider_registry.py, api/models.py) that still described the frontend
table as a manual sync point.
Closes#1082
source_insight was the only content table without created/updated field
definitions. Since the table is SCHEMAFULL and insights are created via a
raw CREATE (create_insight command), SurrealDB silently dropped any
timestamps - rows genuinely held NONE - and the API then wrapped them in
str(), so clients received the literal string "None".
- Migration 19 defines created/updated on source_insight with the same
time::now() defaults used by source, note and notebook, so new insights
are stamped at creation. Existing rows are left untouched (no backfill).
- SourceInsightResponse.created/updated are now Optional[str]; both
routers emit an ISO 8601 string when the timestamp is present and null
when it is absent (legacy rows), never "None".
- Frontend types updated to string | null accordingly.
- New tests cover the migration definition/registration and the API
serialization (absent -> null, present -> ISO string).
Note: Python-side stamping alone was not viable - the table is SCHEMAFULL,
so undefined fields written by the client are silently dropped on
SurrealDB 2.x (verified against a live instance), hence the schema
migration mirroring the other tables.
Closes#1045
PUT /api/models/defaults used 'is not None' guards, so an explicit null
sent to clear a default was silently ignored — the old value survived
while the client saw success (same anti-pattern fixed for credentials
in #1046). The handler is now keyed on field presence (model_fields_set):
absent keeps the current value, explicit null clears it. The required
defaults (chat, embedding) reject null with a 400.
In the UI, the optional default-model selects (transformation, tools,
large context, TTS, STT) now offer a clear option — labeled 'Use
fallback (chat default)' for transformation/tools, which fall back to
the chat default when unset, and 'None' otherwise — and display that
state when no model is assigned. The transformation default is no
longer marked required, matching the backend fallback behavior.
Closes#1091
Consolidate the three copies of context assembly into
open_notebook/utils/context_builder.py:
- POST /api/chat/context now delegates to build_notebook_context()
(same request/response shapes, same string-matching config semantics)
- The source-chat graph now calls build_source_context() instead of the
495-line generalized ContextBuilder class, which had exactly one
caller and whose notebook/notes/priority-config flexibility was dead
- POST /api/notebooks/{notebook_id}/context removed: it duplicated
/api/chat/context with a slightly different envelope and had zero
callers (frontend, docs, tests)
Behavior is pinned by new characterization tests written before the
refactor (tests/test_context_endpoint_characterization.py) plus unit
tests for build_source_context.
The routers wrapped endpoint bodies in a broad 'except Exception' that
re-raised everything as HTTPException(500), intercepting the typed
open_notebook.exceptions hierarchy before the global handlers in
api/main.py could map it to its documented status code (NotFoundError
-> 404, InvalidInputError -> 400, ConfigurationError -> 422,
RateLimitError -> 429, NetworkError/ExternalServiceError -> 502).
Every endpoint-level handler chain across the 18 affected routers now
re-raises HTTPException and OpenNotebookError before the final generic
arm, which keeps catching untyped exceptions and returning a sanitized
500 (the no-raw-exception-text guarantee is unchanged and covered by
tests/test_error_message_sanitization.py). Nested best-effort blocks
(fallbacks, SSE error events, per-item loops) are deliberately left
untouched. In create_source and _create_source_async_path the new
re-raise arms preserve the uploaded-file / orphaned-source cleanup.
Characterization tests that pinned the old wrapped-500 behavior for
source-chat sessions with a missing refers_to relation are updated to
the correct 404, and a new test module asserts per router that a
domain-layer ConfigurationError maps to 422 instead of 500.
* ci: gate PRs on mypy, start ignore_errors burn-down
Add a backend-typecheck CI job running uv run python -m mypy . and bring
the repo-wide baseline from 197 errors to 0 so the gate blocks new type
errors from now on:
- enable the pydantic mypy plugin (resolves 138 false positives on
models whose fields have Field(None, ...) defaults)
- fix the remaining errors with real annotations; the only new type:
ignore comments cover a genuine langgraph typing limitation (partial
state dicts are valid at runtime but the overloads require the full
state type) and tests that intentionally pass invalid input
- start the ignore_errors burn-down: open_notebook.graphs.transformation,
open_notebook.graphs.ask and api.routers.models are now type-checked;
stale blocks for the deleted api.client and api.podcast_api_service
modules removed. Only open_notebook.domain.notebook remains exempt
(DB layer is migrating to surreal-basics)
- fix the mypy.ini header comment to describe what the config does
* test: use typing.get_args on the registry literal check
The registry test landed in parallel using Literal.__args__, which the
mypy gate in this branch rejects; align it with the get_args() idiom
used by the rest of the file.
* refactor(ai): single provider registry as the backend source of truth
Provider metadata (env vars, modalities, connection-test models,
OpenAI-compatible discovery URLs, display names, docs links) is now
defined once in open_notebook/ai/provider_registry.py. The existing
surfaces are derived from it, keeping every import and call-site shape
unchanged:
- api/credentials_service.py: PROVIDER_ENV_CONFIG, PROVIDER_MODALITIES
and the discovery url_map are built from the registry
- open_notebook/ai/connection_tester.py: TEST_MODELS derived
- open_notebook/ai/model_discovery.py: OPENAI_COMPAT_PROVIDERS built
from registry entries with a discovery URL (quirk hooks stay local)
The SupportedProvider Literal (typing, can't be built at runtime) and
the frontend provider tables remain manual copies; the cross-check
tests now assert registry keys == Literal == frontend list, plus
registry internal consistency and discovery-table coverage.
New GET /api/providers endpoint exposes the registry (name, display
name, modalities, docs_url, env-configured status) so clients can stop
hardcoding provider lists (frontend adoption is a follow-up).
Docs updated: open_notebook/AGENTS.md and docs/7-DEVELOPMENT/credentials.md
now describe the registry instead of the four-place sync rule.
* refactor(ai): address review findings on the provider registry
- Build PROVIDERS via _build_registry(), which raises on a duplicate
provider name at import time instead of silently dropping the earlier
spec (dict-comprehension behavior); regression test added
- Pin the exact OpenAI-compatible provider -> discovery URL mapping in
a test so a registry edit can't silently drop or misassign a URL
- Give TEST_MODELS a real type annotation
(Dict[str, Tuple[Optional[str], str]]) instead of bare dict
- Extract _source_to_response() shared builder (SourceResponse was
hand-rolled 5x with the same nested AssetModel ternary)
- Extract _cleanup_uploaded_file() (the unlink ritual was pasted 6x
inside create_source); inner cleanups were redundant with the outer
exception handlers, which always run for the same failures
- Split create_source into _build_content_state() (type validation +
SSRF/LFI guards) and _create_source_async_path() /
_create_source_sync_path()
- Unify the paginated list query (with/without notebook filter differed
only in the FROM clause and bound params)
Pure structural refactor: same status codes, error messages and
response shapes. Security checks (atomic filename claim via
touch(exist_ok=False), path-traversal containment, SSRF/LFI guards)
preserved verbatim, with their comments.
* test(api): characterize shared chat/source-chat router behaviors
* refactor(api): extract shared session and message helpers for chat routers
* refactor(api): mark intentionally unused source id unpacks
The session handlers only need the verified session; the source-level
verification happens inside get_verified_source_session. Underscore the
unused binding in get/update/delete to make that explicit.
Collapse the eight structurally identical OpenAI-compatible provider
discovery functions (openai, groq, mistral, deepseek, xai, openrouter,
dashscope, minimax) into one generic discover_openai_compatible_provider()
driven by an OPENAI_COMPAT_PROVIDERS spec table. Per-provider quirks
(Mistral capability flags, OpenRouter fixed type + description) are kept
as optional hooks on the spec. Module-level function names and
PROVIDER_DISCOVERY_FUNCTIONS keys are preserved.
Replace the stale hardcoded Anthropic model list (claude-3 era) with
real discovery via GET https://api.anthropic.com/v1/models (paginated,
x-api-key + anthropic-version headers) — the comment claiming Anthropic
has no listing API was wrong. Keep a refreshed static fallback
(ANTHROPIC_FALLBACK_MODELS, current Claude 4.x/5 aliases) for when the
API call fails, and reuse the same fetch-with-fallback in the
credential-based discovery path in api/credentials_service.py, which
carried a second copy of the stale list.
Add tests covering the table-driven path, provider quirks, Anthropic
pagination/headers, and the fallback behavior (all HTTP mocked).
Remove F401 (unused imports), F841 (unused local variables) and E722
(bare except) from the ruff ignore list; the legacy Streamlit-era code
that motivated ignoring them is gone.
Fallout fixed: 10 unused imports removed (none were load-bearing
re-exports or side-effect imports), 2 unused mock bindings in tests
dropped, and a now-empty TYPE_CHECKING block in
open_notebook/utils/embedding.py cleaned up. No bare excepts remained
in the codebase.
Delete api/client.py (synchronous httpx client calling the app's own
FastAPI server, a leftover from the removed Streamlit UI) and the 13
api/*_service.py wrappers around it. None of these files had any
importer in routers, commands or tests, verified by grepping the whole
repo for each module name.
Also remove commands/example_commands.py (process_text/analyze_data
demo commands from the surreal-commands README) and its export from
commands/__init__.py, and update remaining docstring examples to
reference the real generate_podcast command.
The real services (command_service, credentials_service,
podcast_service) are untouched.
* fix(frontend): actually clear credential fields when emptied in the edit dialog
Clearing base_url (or the Vertex project/location/credentials_path)
in the credential edit dialog silently did nothing: the submit handler
mapped an emptied field to `undefined`, JSON.stringify dropped the key
from the PUT body, and the backend's partial-update semantics kept the
old value — while the UI reported success.
Emptied fields are now sent as an explicit `null`, which the API
already accepts and persists as a cleared value (verified live).
UpdateCredentialRequest's nullable fields are typed accordingly.
Found in v1.11 release testing (pre-existing, not a release regression):
an Ollama credential with a stale IP in base_url could not be cleared
from the UI.
* fix(api): clear credential fields on explicit null, not just empty string
The update handler guarded every field with 'is not None', so a JSON
null sent to clear base_url (or endpoint/api_version/endpoint_*/
project/location/credentials_path/num_ctx) was silently skipped — the
old value survived while the client saw a 200. Combined with the
frontend bug fixed in the previous commit (emptied fields dropped from
the payload entirely), clearing a credential field was impossible from
the UI: during release testing an Ollama credential kept pointing at a
stale IP after being 'cleared', and only an empty-string PUT crafted by
hand actually cleared it.
Field updates are now keyed on presence in model_fields_set: absent
keeps the old value, explicit null or "" clears. name/modalities/
api_key keep their non-null guards (clearing those is not meaningful).
Regression tests cover null-clears, empty-string-clears, absent-keeps
and the Vertex credentials_path case.
* test(frontend): extract credential update payload builder and cover clear-on-empty
Addresses the cubic review: the frontend half of the fix had no
reproducing test. The edit dialog's payload construction now lives in
a pure buildCredentialUpdatePayload() (frontend/src/lib/
credential-update-payload.ts) used by the component, with tests that
fail on the original bug: an emptied base_url (and Vertex project/
location/credentials_path) must survive JSON.stringify as an explicit
null, unchanged fields must be omitted, and emptied num_ctx clears
via 0.
Two findings from v1.11 release testing:
- GET /api/sources?sort_by=title returned a 500. source.title carries a
SEARCH (BM25) index (idx_source_title, migration 1) and SurrealDB's
planner fails ORDER BY on such a column with 'No iterator has been
found'. The query now sorts by a computed alias
(string::lowercase(title OR '') AS title_sort), which sidesteps the
index and makes the sort case-insensitive as a bonus.
- POST /api/sources with an over-limit notebooks/transformations array
(or invalid JSON in either field) returned a raw 500.
parse_source_form_data() builds SourceCreate manually, so pydantic's
ValidationError never reached FastAPI's request-validation handler.
Both cases now surface as a clean 422 with a descriptive message.
Verified against a live SurrealDB v2 instance; regression tests added
for the ORDER BY alias, all six sort fields, and the 422 paths.
* fix: make the provider connection test resilient to model retirement
Two independent hardenings against the #970 class of breakage (a hard-coded
Gemini test model getting shut down by Google, which made testing a valid
key fail with a 404):
- Use Google's floating alias gemini-flash-latest for the Google/Vertex
test model instead of a dated id, so a retirement repoints it for us.
- Reframe the provider connection test around what an error actually
proves: only a rejected key (401), missing permissions (403), or an
unreachable endpoint are failures. Anything the provider returns after
authenticating - a rate limit, or a missing/retired/unsupported model -
still proves the credentials work, so it reports success. Previously this
relied on matching the literal phrase 'not found' + 'model', which a
differently-worded retirement/deprecation error slipped past.
Unifies the auth/network/rate-limit classification (previously duplicated
and divergent between connection_tester and credentials_service) into
shared helpers. The individual-model test keeps model-not-found as a
failure, since there a specific registered model really is broken.
Adds classification tests with realistic provider error strings.
* docs: add CHANGELOG entry for connection-test resilience fix
* fix: update Google model version in connection tester and tests
* fix: updated the gemini model lists that references deprecated models
* fix: update also documentation. This is more prone to not follow the maintainer's directives, so PTADL
* fix: forgot two references to gemini deprecated versions
* fix: use valid, longer-lived gemini model IDs
Several IDs the PR introduced don't exist or are near shutdown, verified
against Google's official model/deprecation pages (2026-07):
- connection test model gemini-2.5-flash -> gemini-3.5-flash (2.5-flash
retires 2026-10-16; 3.5-flash is the current stable GA and Google's
named replacement, so the #970 fix doesn't re-break in ~3 months)
- gemini-3.5-pro (does not exist) -> gemini-2.5-pro in docs
- plain gemini-3.1-flash (not a GA Gemini-API id) dropped from the Vertex
discovery list / preferences / docs; use gemini-3.5-flash or -flash-lite
- dead gemini-pro dropped from preferences and docs
Ported #996's #970 regression test, pinned to gemini-3.5-flash.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
check_api_password() (an unused HTTPBearer-based dependency, superseded
by PasswordAuthMiddleware) and its now-unused imports are dead code -
nothing calls it. Removed.
Docs across api/CLAUDE.md, docs/3-USER-GUIDE/api-configuration.md,
docs/5-CONFIGURATION/security.md, docs/7-DEVELOPMENT/security.md, and
docs/SECURITY_REVIEW.md still described a hardcoded default password
("open-notebook-change-me") that PasswordAuthMiddleware doesn't actually
have - if OPEN_NOTEBOOK_PASSWORD is unset, auth is fully disabled
instead. Updated to match actual behavior.
* docs: restructure documentation around AGENTS.md, VISION.md and decision records
- Consolidate 17 CLAUDE.md files into 3 AGENTS.md (root, backend, frontend);
CLAUDE.md files become @AGENTS.md pointers
- Add VISION.md: product identity + current posture with horizon clusters
- Add docs/7-DEVELOPMENT/decisions/ with 4 retroactive ADRs and 2 PDRs
- Add 5 new engineering docs pages (credentials, content-processing,
podcasts, prompts, frontend) absorbing knowledge from removed CLAUDE.md
- Dismember TRIAGE.md: label taxonomy into maintainer-guide.md, product
jurisprudence into VISION.md, operator heuristics stay local (gitignored)
- Add AI-assisted/agent-generated PR guidelines to contributing.md
- Convert README.dev.md into a pointer after migrating its unique content
(make workflow matrix, Docker publishing, add-a-language playbook)
- Fix stale docs: migration path/format, provider count, locale list;
fix broken links (docs/index.md, PR template, CONFIGURATION.md)
* docs: fix README doc links and add markdown link check to CI
- Repoint 9 README links to pages that actually exist in docs/
- Replace literal (link) placeholder in maintainer-guide templates
- Add scripts/check_md_links.py validating relative links in tracked
markdown (skips URLs, anchors and code spans)
- Add docs-links workflow running the check on PRs that touch markdown
* docs: add documentation restructure to changelog
* feat: add cubic.yaml with project-aware AI review agents
Three custom review agents (vision & principles alignment backed by
VISION.md, known mechanical caveats, security & testability), PR-contract
review instructions, and automatic ultrareviews for auth, credential,
encryption and migration changes.
* docs: graduate issue-first policy by change size
Small obvious fixes (typos, docs, tiny bugs, i18n completions) no longer
require an issue; features and architecture changes still do. Sizeable
PRs opened without an issue convert to draft while the issue goes
through triage (1-2 days). Applied consistently across contributing
guide, root CONTRIBUTING pointer, PR template, maintainer guide red
flags and cubic review instructions.
* docs: align PR template Related Issue section with graduated issue-first policy
* docs: address review — generalize ADR-002/004, unwrap hard-wrapped lines
- ADR-002 now records the general delegation rule (platform/media support
that needs heavy coding lives in focused external libraries) covering
Esperanto, Content Core and podcast-creator
- ADR-004 now records the durable decision (long-running work runs on
background workers — heavy content, varied machine sizes, never lock
usage) with the queue technology as a swappable implementation detail
pending #381
- Remove mid-paragraph hard line wrapping from authored docs to match
repo convention (one line per paragraph)
* fix: address cubic review — stale doc facts, make dev/full targets, link checker query strings
- credentials.md: only PROVIDER_CONFIG exists as a map; Vertex/Azure/
OpenAI-compatible provisioning is inline in _provision_*() functions
- content-processing.md: correct ContextConfig priority weights
(source 100 > insight 75 > note 50)
- development-setup.md + Makefile: make dev/full pointed at root compose
files that don't exist; targets now use examples/docker-compose-dev.yml
and examples/docker-compose-full-local.yml with --project-directory .
- check_md_links.py: strip query strings before file-existence checks
* fix: surface silent command-submission failures where they matter
Source.add_insight() caught submission failures and returned None
instead of raising - callers (transformation.py, source.py) run inside
surreal-commands jobs whose outer exception handling already
retries/fails on this, so a swallowed submission failure meant a
transformation could report success while the insight was silently
never persisted. Now raises DatabaseOperationError, matching
vectorize()'s existing contract.
Note.save()'s auto-embed needs the opposite treatment: it's an implicit
side effect of save() (not an explicit dedicated call), and the note
itself is already durably saved by the time it runs - so a submission
hiccup there shouldn't turn an otherwise-successful save into a 500.
Wrapped in try/except, logs and returns None. api/routers/embedding.py's
explicit POST /embed (item_type=note) is the one caller for whom
submission success genuinely is the point of the call, so it separately
checks for a missing command_id and surfaces that as a failure.
* fix: collapse duplicate error logging in add_insight
logger.error + logger.exception logged the same failure twice; a single
logger.exception call carries both the message and the traceback (same
form vectorize() already uses).
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Two small fixes to the podcast episode-listing path, plus a doc note:
- audio_file is only ever set server-side today from a UUID-named
directory under PODCASTS_FOLDER, so this can't currently be tripped -
but the stream/retry/delete endpoints didn't verify the resolved path
actually stayed within PODCASTS_FOLDER before following it. Add
_is_audio_path_contained() as defense in depth against a future code
path (e.g. importing external audio) setting audio_file to something
else.
- Listing episodes called get_job_detail() -> get_command_status() once
per episode, each its own round trip (no connection pooling). Add
PodcastEpisode.get_job_details_for_commands() to batch-fetch status
for every episode's command in one query instead.
Also documents (docs/7-DEVELOPMENT/security.md) that podcast_creator's
configure("templates", {...}) compiles strings as Jinja2 template
source - the same shape as the SSTI vulnerability fixed in
transformation.py (GHSA-f35w-wx37-26q7). Confirmed dormant: no code path
in this repo calls it today. commands/podcast_commands.py gets a
matching code comment warning against wiring user text into it if a
"custom podcast template" feature is ever added.
api/routers/sources.py and api/podcast_service.py interpolated the raw
exception (detail=f"...: {str(e)}") into client-facing error responses -
inconsistent with the safer pattern already used elsewhere in the same
files (e.g. the download handlers), which log the raw exception
server-side but return a fixed generic message. Internal details (DB
hostnames, connection errors, stack-trace fragments) could leak to any
API caller through a 500 response.
Every occurrence already had a matching logger.error() call, so this is
a client-facing message change only, not a logging change. Deliberate
app-authored messages (InvalidInputError text, "Notebook X not found",
result.error_message) are untouched - only raw f"{str(e)}" interpolation
is affected.
tests/test_config_endpoint_no_leak.py is a regression lock for
api/routers/config.py (already correct, not touched by this diff) rather
than a fix - added since it shares the same "unauthenticated endpoint,
don't leak exception text" concern.
* fix: restrict CreateCredentialRequest.provider to a known allowlist
provider was a bare `str`, so any string (typo'd or bogus) flowed
through validation to the domain layer and failed later with a less
clear error, instead of a clean 422 at the API boundary.
Add a SupportedProvider Literal covering the 17 providers already
handled elsewhere - kept in sync with the frontend's ALL_PROVIDERS,
connection_tester.py's TEST_MODELS, and credentials_service.py's
PROVIDER_ENV_CONFIG (a test asserts all three agree on the same set).
* test: lock the frontend provider list into the sync test
The frontend's ALL_PROVIDERS copy was only cross-checked in a docstring;
extract its string literals from the source so all four provider lists
(Literal, TEST_MODELS, PROVIDER_ENV_CONFIG, frontend) are enforced by CI.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Three small, independent hardening fixes to source ingestion:
- generate_unique_filename() checked `if not resolved.exists()` then let
a separate write happen later - two concurrent uploads landing on the
same candidate name could both pass the check and clobber each other.
Now atomically claims the name via Path.touch(exist_ok=False) (O_EXCL)
as part of the search loop itself.
- _resolve_source_file() and _is_source_file_available() compared
`resolved_path.startswith(safe_root)` without a trailing separator - a
sibling directory that merely starts with the same string (e.g.
"uploads_evil/") would incorrectly be treated as contained, unlike this
file's other two path checks which already guard with `+ os.sep`. Not
reachable today (source.asset.file_path is only ever set server-side),
but this closes the gap and matches the existing pattern.
- SourceCreate.notebooks/transformations had no length limit; both are
iterated with a per-item DB lookup in create_source(), so an unbounded
array let a single request trigger an unbounded number of sequential DB
round trips. Capped at 50.
tests/test_upload_type_mitigations.py adds no code change - it documents
why the adjacent "no file type allowlist on uploads" finding was
investigated and judged low-risk without one (downloads are already
served as application/octet-stream regardless of actual file type).
* fix: reject oversized request bodies before auth/routing
No limit existed on request body size, so a single upload could exhaust
memory/disk before any validation ran. MaxBodySizeMiddleware
(api/middleware.py) rejects requests over OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB
(default 100MB) via both a Content-Length pre-check and by counting bytes
as the body streams in, so it also catches chunked requests with no
Content-Length header.
Registered after PasswordAuthMiddleware (wrapping it) so oversized
requests are rejected before spending any work on credential checks, and
inside CORSMiddleware so a rejected upload still gets CORS headers.
* fix: clamp non-positive size limits and log rejected requests
OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB=0 (or negative) made every request with a
body 413; fall back to the default instead, with a warning. Also log each
413 rejection so operators can diagnose failed uploads.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: don't combine wildcard CORS origins with allow_credentials
Combining allow_origins=["*"] with allow_credentials=True makes
Starlette's CORSMiddleware reflect the request's Origin header verbatim
instead of returning a literal "*" (browsers reject a literal wildcard
alongside credentials) - defeating the origin allowlist for any
credentialed request.
allow_credentials is now tied to whether CORS_ORIGINS was explicitly
scoped: False for the default wildcard, True once an operator opts into
specific origins. _cors_headers() (the manual CORS builder for error
responses raised before CORSMiddleware runs) is updated to match, so it
can't grant credentials the real middleware wouldn't.
Not independently exploitable today (the frontend never sends
credentialed requests, and auth is a Bearer header, not a cookie), but
there's no reason to allow it for the default wildcard case.
* fix: key allow_credentials on the parsed origins list, not the env var
An operator who explicitly sets CORS_ORIGINS=* got allow_credentials=True
with a wildcard origin list - the exact reflect-any-Origin behavior this
change exists to prevent. Introduce CORS_ALLOW_CREDENTIALS keyed on the
parsed list containing '*' and use it at both the middleware registration
and the manual error-response headers; replace the tautological formula
tests with ones exercising the real parser.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: don't leak filesystem info via Vertex credential test errors
Vertex's credentials_path is free text with no path validation, and
Google's auth library raises distinguishable exceptions for "file
missing" (OSError), "not valid JSON" (json.JSONDecodeError), and "valid
JSON but wrong shape" (GoogleAuthError). Echoing any of these back to an
API caller turns credential/model testing into a filesystem oracle - an
attacker who can create/test a Vertex credential could probe for the
existence and contents-shape of arbitrary files on the server.
_is_vertex_credentials_file_error() catches all three and both call
sites (api/credentials_service.py's test_credential, connection_tester's
test_individual_model) return one generic message instead.
* fix: don't classify network failures as credentials-file errors
ConnectionError/TimeoutError are OSError subclasses and TransportError is
a GoogleAuthError subclass, so the oracle guard also swallowed genuine
network failures - a user with a blocked network would be told to go debug
their credentials file. Excluding them reveals only the error's category,
which keeps the filesystem oracle closed.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: sanitize raw HTML in the note/transformation markdown preview
MarkdownEditor's live preview renders through @uiw/react-markdown-
preview, which parses literal HTML in the markdown source into real
elements (its `raw` default) - including a live <iframe>. Notes can
hold AI-generated content that echoes an indirect prompt injection
from an ingested document, so this was reachable without the user
writing any HTML themselves.
Add rehype-sanitize (default schema) ahead of rehype-katex in the
preview pipeline. Ordering matters: sanitizing after katex strips
katex's own generated markup (not in the default allowlist), while
sanitizing before it only touches the raw-HTML-derived tree and
leaves not-yet-rendered math nodes alone. Verified against the real
preview component that this strips <iframe>/<script>/<style>/
javascript: URLs while fully preserving math, syntax highlighting,
and GFM tables/task-lists.
* fix: make validate_url() async so DNS resolution doesn't block the event loop
socket.getaddrinfo() in validate_url()'s hostname-resolution branch ran
synchronously on every call - and this is called on the hot path of model
provisioning, potentially once per chat message/transformation. A slow or
hanging DNS lookup stalled every other concurrent request, not just the
one that triggered it.
Run the resolution via asyncio.to_thread() instead, and thread `await`
through every call site: credentials_service.py's discover functions,
routers/credentials.py's create/update, routers/sources.py's source-URL
ingestion, and connection_tester.py's three provider test functions.
save_uploaded_file() did a plain synchronous open()/write() directly
in the async create_source handler, blocking the event loop - and
every other concurrent request - for the duration of a large upload.
Same bug class the recent chat-graph fix (#971) addressed, just not
applied here.
Move the filesystem work (filename resolution + write) into a sync
helper run via asyncio.to_thread(), matching the pattern already used
for execute_command_sync elsewhere in this file. Confirmed the event
loop stays responsive (ticking normally) during a simulated slow
write, and that errors/cleanup still propagate correctly through the
thread.
Building default chat context, the notebook context endpoint, and
podcast generation all looped over every source in a notebook calling
get_context() -> get_insights(), each a separate query that also pays
its own connection setup (no pooling). A notebook with hundreds of
sources meant hundreds of serialized round trips before a chat
message even reached the LLM.
Add SourceInsight.get_for_sources() to fetch insights for every
source in one query, and thread an optional pre-fetched insights list
through Source.get_context() so callers can opt in without changing
its behavior for anyone who doesn't. Measured against a real
(embedded) SurrealDB instance: 14 queries down to 3 for a 12-source
notebook, with correctness verified. The two router call sites treat
a batch-fetch failure the same way the old per-source loop treated a
single failure - falls back to empty insights rather than failing the
whole request.
validate_url() only ran when a credential was created or updated. The
actual HTTP requests (connection testing, model discovery, and real
inference through Esperanto) re-resolve DNS fresh on every call, so a
hostname that resolved to a public IP at save time can later be
repointed to an internal or cloud-metadata address - a classic
DNS-rebinding TOCTOU that a one-time check can't catch.
Move validate_url() to open_notebook/utils so it can be re-run from
the AI layer without an api-depends-on-open_notebook layering
violation, and re-check immediately before every outbound request:
connection_tester's three providers, credential model discovery, and
ModelManager.get_model() on the real-inference path. Also close a
second gap in the same validator: it missed AWS's IPv6 metadata
address (fd00:ec2::254), which isn't link-local so the existing check
never caught it.
POST /sources with type=link copied the user-supplied URL straight
into content-core's fetch with zero validation - unlike the
credential-URL path, which already blocks internal/metadata
addresses. Any user could make the server fetch cloud metadata
endpoints or scan the internal network via "add a web source".
Reuse the same validate_url() guard at the point the URL is first
accepted, before it's ever handed to content-core.
PasswordAuthMiddleware and check_api_password compared the bearer
token to the configured password with `!=`, a timing side-channel on
the single secret gating the whole API. There's no rate limiting to
blunt repeated probing, so switch both to secrets.compare_digest().
repo_relate() interpolated the relate target directly into the query
string, reachable via an unvalidated notebook_id on the save-insight-
as-note flow - a crafted ID could inject and execute arbitrary
SurrealQL (confirmed against a live embedded instance: a single
crafted RELATE call wiped an entire table). Bind record identifiers
as query parameters instead of building them into the query text, and
validate notebook_id exists before relating, matching the pattern
already used by every other caller of add_to_notebook().
* fix: make API startup resilient to a not-yet-ready database
Fixes#708
* fix: bound the database readiness probe with a per-attempt timeout
_wait_for_database awaited migration_manager.ping() with no timeout, so a
hung connection probe could bypass the retry budget and block startup
indefinitely. Wrap the probe in asyncio.wait_for with a per-attempt ceiling;
a timed-out probe is treated as a transient failure and retried like any
other unreachable-database attempt, keeping the retry budget bounded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for startup database readiness retry (#708)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* feat: Recently Viewed section for notebooks and sources
Fixes#850
* fix: export RecentlyViewedResponse type from api types
The RecentlyViewed component and notebooks API client import
RecentlyViewedResponse from @/lib/types/api but the interface was missing,
failing the type-check build. Add it to match the backend api/models.py model.
* fix: harden recently-viewed read path and index the recency query
Three robustness fixes on the recently-viewed feature:
- Make the last_viewed_at write-on-read stamping best-effort. _stamp_source_view
and _stamp_notebook_view now swallow and log their own errors, so a failed
stamp update can no longer turn a successful GET /sources/{id} or
GET /notebooks/{id} into a 500.
- Stop leaking internal details from GET /recently-viewed: log the full
exception server-side and return a generic error message to the client.
- Add indexes on last_viewed_at for the notebook and source tables in
migration 16 so the ORDER BY last_viewed_at DESC LIMIT recently-viewed
query does not degrade into a full table scan as data grows. The down
migration removes the indexes before dropping the field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: register migration 18, add recently-viewed i18n keys and CHANGELOG entry
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* feat: per-transformation custom model selection
Fixes#776
* fix: validate model_id exists when creating or updating a transformation
The create and update endpoints persisted model_id without checking the
referenced model exists, so an invalid reference was stored silently and only
surfaced later as a 404 at execution time. Add the same existence check
execute_transformation already performs (Model.get -> 404 if missing) to both
the create and update paths, so a bad model_id is rejected up front with a
clear 404. Update still allows clearing model_id to None.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: cover model_id existence validation on transformation create/update
The create test now mocks Model.get so the new existence check passes and
asserts it was awaited with the supplied model_id. The update test expects
Model.get to be awaited twice (once validating the update, once at execute)
rather than once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for per-transformation model selection (#776)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: pass max_tokens through to podcast_creator for outline/transcript generation
* fix: persist and expose episode_profile max_tokens (migration + API)
* docs: add CHANGELOG entry for episode profile max_tokens pass-through (#639)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Add type annotations to PasswordAuthMiddleware.__init__ and dispatch (plus
excluded_paths) and drop the [mypy-api.auth] ignore_errors stanza so the
module is type-checked. Annotations only; no behavior change.
Refs #945
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Expand /api/sources sorting to type, title, created, updated, insights_count, and embedded (whitelisted, with a computed type alias so SurrealDB can order by it). Frontend adds sortable headers for every column plus an Updated column.
Fixes#895
Notebook chat (execute_chat) and source-chat SSE ran LangGraph's synchronous invoke() on the event loop, freezing the whole API while the LLM responded. Both now run via asyncio.to_thread(), matching the existing get_state() calls.
Fixes#704
Switches the streaming endpoints (/api/search/ask, source chat messages) to media_type=text/event-stream and adds dedicated App Router SSE route handlers (_sse-proxy.ts) that stream the upstream body directly, bypassing the Next.js rewrites proxy's gzip buffering in standalone/production mode.
Verified locally: with Accept-Encoding: gzip (i.e. a real browser), the standalone rewrites proxy buffered the entire SSE stream to completion; the route-handler approach restores progressive delivery.
* fix(api): return 404 instead of 500 for missing resources in CRUD endpoints (#862)
ObjectModel.get() raises NotFoundError for a missing record (never returns a
falsy value), so the 'if not obj: 404' guards were dead code and each handler's
broad 'except Exception' re-raised NotFoundError as 500, never hitting the global
NotFoundError->404 handler.
Add an explicit 'except NotFoundError -> 404' arm to the affected handlers in
notebooks, notes, models, credentials and embedding routers, plus regression
tests asserting 404 (not 500) when .get() raises NotFoundError.
* refactor(api): remove dead 'if not obj' guards after .get() (cubic #924)
Now that each handler maps NotFoundError -> 404, the 'if not obj: raise 404'
guards are unreachable (.get() raises rather than returning a falsy value).
Drop them; where the fetched object was only used by the guard, keep the .get()
call for its existence-validation side effect without binding an unused var.
* chore(release): prepare v1.10.0
* style: fix import ordering flagged by ruff
* fix(sources): return 404 for missing source and fix retry 500 from double-prefixed command id
- GET /sources/{id} mapped NotFoundError to a generic 500; now returns 404
- POST /sources/{id}/retry double-prefixed the command id (command:command:...),
raising 'too many values to unpack' after queuing; align with the create path
- update retry test mock to realistic prefixed command id + guard against
double-prefix; add 404 regression test
* fix(sources): mark failed extraction as failed so retry surfaces (#726)
content-core signals soft failures (unreachable/invalid URL) by returning
title=Error + 'Failed to extract content:' body instead of raising, and the
process_source command swallowed permanent ValueErrors into a success=False
result. Since surreal-commands marks a job completed when the function returns,
failed ingests showed status 'completed' and never offered the retry button.
- source graph: detect the content-core failure sentinel and raise
- process_source_command: re-raise ValueError (stop_on already prevents retry)
so the job is marked failed and the source becomes retryable
* feat(notebook): per-type bulk context actions for sources and notes (#223)
- Sources context menu now offers 'insights only' (sources without insights
are excluded rather than forced to full), 'full content', and 'exclude all'
- Add the same bulk Context menu to the Notes column (include all / exclude all)
- Bulk choices propagate to items loaded later via pagination
- New locale keys (includeAllInsights/includeAllFull) across all 14 locales
- Unit tests for the new bulk modes and note context helpers
* fix(search): reject non-positive limit and survive highlight overflow
- SearchRequest.limit now has ge=1 so 0/negative values are rejected with
422 instead of flowing into SurrealDB as LIMIT -1/0 (500 or empty set) (#863)
- text_search() catches SurrealDB's "position overflow" highlight error and
falls back to vector_search() so large/multi-byte chunks no longer 500 (#648)
- add tests covering limit validation and the overflow fallback path
* fix(search): surface error when vector fallback also fails
Address cubic review on #898:
- text_search() no longer returns [] when the vector-search fallback also
fails; it logs the traceback and raises DatabaseOperationError so a total
search outage is not masked as a legitimate empty result set
- document the fallback + failure behavior in domain/CLAUDE.md
- update the test to assert the double-failure path raises
