* fix(frontend): actually clear credential fields when emptied in the edit dialog
Clearing base_url (or the Vertex project/location/credentials_path)
in the credential edit dialog silently did nothing: the submit handler
mapped an emptied field to `undefined`, JSON.stringify dropped the key
from the PUT body, and the backend's partial-update semantics kept the
old value — while the UI reported success.
Emptied fields are now sent as an explicit `null`, which the API
already accepts and persists as a cleared value (verified live).
UpdateCredentialRequest's nullable fields are typed accordingly.
Found in v1.11 release testing (pre-existing, not a release regression):
an Ollama credential with a stale IP in base_url could not be cleared
from the UI.
* fix(api): clear credential fields on explicit null, not just empty string
The update handler guarded every field with 'is not None', so a JSON
null sent to clear base_url (or endpoint/api_version/endpoint_*/
project/location/credentials_path/num_ctx) was silently skipped — the
old value survived while the client saw a 200. Combined with the
frontend bug fixed in the previous commit (emptied fields dropped from
the payload entirely), clearing a credential field was impossible from
the UI: during release testing an Ollama credential kept pointing at a
stale IP after being 'cleared', and only an empty-string PUT crafted by
hand actually cleared it.
Field updates are now keyed on presence in model_fields_set: absent
keeps the old value, explicit null or "" clears. name/modalities/
api_key keep their non-null guards (clearing those is not meaningful).
Regression tests cover null-clears, empty-string-clears, absent-keeps
and the Vertex credentials_path case.
* test(frontend): extract credential update payload builder and cover clear-on-empty
Addresses the cubic review: the frontend half of the fix had no
reproducing test. The edit dialog's payload construction now lives in
a pure buildCredentialUpdatePayload() (frontend/src/lib/
credential-update-payload.ts) used by the component, with tests that
fail on the original bug: an emptied base_url (and Vertex project/
location/credentials_path) must survive JSON.stringify as an explicit
null, unchanged fields must be omitted, and emptied num_ctx clears
via 0.
Two findings from v1.11 release testing:
- GET /api/sources?sort_by=title returned a 500. source.title carries a
SEARCH (BM25) index (idx_source_title, migration 1) and SurrealDB's
planner fails ORDER BY on such a column with 'No iterator has been
found'. The query now sorts by a computed alias
(string::lowercase(title OR '') AS title_sort), which sidesteps the
index and makes the sort case-insensitive as a bonus.
- POST /api/sources with an over-limit notebooks/transformations array
(or invalid JSON in either field) returned a raw 500.
parse_source_form_data() builds SourceCreate manually, so pydantic's
ValidationError never reached FastAPI's request-validation handler.
Both cases now surface as a clean 422 with a descriptive message.
Verified against a live SurrealDB v2 instance; regression tests added
for the ORDER BY alias, all six sort fields, and the 422 paths.
* fix: make the provider connection test resilient to model retirement
Two independent hardenings against the #970 class of breakage (a hard-coded
Gemini test model getting shut down by Google, which made testing a valid
key fail with a 404):
- Use Google's floating alias gemini-flash-latest for the Google/Vertex
test model instead of a dated id, so a retirement repoints it for us.
- Reframe the provider connection test around what an error actually
proves: only a rejected key (401), missing permissions (403), or an
unreachable endpoint are failures. Anything the provider returns after
authenticating - a rate limit, or a missing/retired/unsupported model -
still proves the credentials work, so it reports success. Previously this
relied on matching the literal phrase 'not found' + 'model', which a
differently-worded retirement/deprecation error slipped past.
Unifies the auth/network/rate-limit classification (previously duplicated
and divergent between connection_tester and credentials_service) into
shared helpers. The individual-model test keeps model-not-found as a
failure, since there a specific registered model really is broken.
Adds classification tests with realistic provider error strings.
* docs: add CHANGELOG entry for connection-test resilience fix
* fix: update Google model version in connection tester and tests
* fix: updated the gemini model lists that references deprecated models
* fix: update also documentation. This is more prone to not follow the maintainer's directives, so PTADL
* fix: forgot two references to gemini deprecated versions
* fix: use valid, longer-lived gemini model IDs
Several IDs the PR introduced don't exist or are near shutdown, verified
against Google's official model/deprecation pages (2026-07):
- connection test model gemini-2.5-flash -> gemini-3.5-flash (2.5-flash
retires 2026-10-16; 3.5-flash is the current stable GA and Google's
named replacement, so the #970 fix doesn't re-break in ~3 months)
- gemini-3.5-pro (does not exist) -> gemini-2.5-pro in docs
- plain gemini-3.1-flash (not a GA Gemini-API id) dropped from the Vertex
discovery list / preferences / docs; use gemini-3.5-flash or -flash-lite
- dead gemini-pro dropped from preferences and docs
Ported #996's #970 regression test, pinned to gemini-3.5-flash.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
check_api_password() (an unused HTTPBearer-based dependency, superseded
by PasswordAuthMiddleware) and its now-unused imports are dead code -
nothing calls it. Removed.
Docs across api/CLAUDE.md, docs/3-USER-GUIDE/api-configuration.md,
docs/5-CONFIGURATION/security.md, docs/7-DEVELOPMENT/security.md, and
docs/SECURITY_REVIEW.md still described a hardcoded default password
("open-notebook-change-me") that PasswordAuthMiddleware doesn't actually
have - if OPEN_NOTEBOOK_PASSWORD is unset, auth is fully disabled
instead. Updated to match actual behavior.
* docs: restructure documentation around AGENTS.md, VISION.md and decision records
- Consolidate 17 CLAUDE.md files into 3 AGENTS.md (root, backend, frontend);
CLAUDE.md files become @AGENTS.md pointers
- Add VISION.md: product identity + current posture with horizon clusters
- Add docs/7-DEVELOPMENT/decisions/ with 4 retroactive ADRs and 2 PDRs
- Add 5 new engineering docs pages (credentials, content-processing,
podcasts, prompts, frontend) absorbing knowledge from removed CLAUDE.md
- Dismember TRIAGE.md: label taxonomy into maintainer-guide.md, product
jurisprudence into VISION.md, operator heuristics stay local (gitignored)
- Add AI-assisted/agent-generated PR guidelines to contributing.md
- Convert README.dev.md into a pointer after migrating its unique content
(make workflow matrix, Docker publishing, add-a-language playbook)
- Fix stale docs: migration path/format, provider count, locale list;
fix broken links (docs/index.md, PR template, CONFIGURATION.md)
* docs: fix README doc links and add markdown link check to CI
- Repoint 9 README links to pages that actually exist in docs/
- Replace literal (link) placeholder in maintainer-guide templates
- Add scripts/check_md_links.py validating relative links in tracked
markdown (skips URLs, anchors and code spans)
- Add docs-links workflow running the check on PRs that touch markdown
* docs: add documentation restructure to changelog
* feat: add cubic.yaml with project-aware AI review agents
Three custom review agents (vision & principles alignment backed by
VISION.md, known mechanical caveats, security & testability), PR-contract
review instructions, and automatic ultrareviews for auth, credential,
encryption and migration changes.
* docs: graduate issue-first policy by change size
Small obvious fixes (typos, docs, tiny bugs, i18n completions) no longer
require an issue; features and architecture changes still do. Sizeable
PRs opened without an issue convert to draft while the issue goes
through triage (1-2 days). Applied consistently across contributing
guide, root CONTRIBUTING pointer, PR template, maintainer guide red
flags and cubic review instructions.
* docs: align PR template Related Issue section with graduated issue-first policy
* docs: address review — generalize ADR-002/004, unwrap hard-wrapped lines
- ADR-002 now records the general delegation rule (platform/media support
that needs heavy coding lives in focused external libraries) covering
Esperanto, Content Core and podcast-creator
- ADR-004 now records the durable decision (long-running work runs on
background workers — heavy content, varied machine sizes, never lock
usage) with the queue technology as a swappable implementation detail
pending #381
- Remove mid-paragraph hard line wrapping from authored docs to match
repo convention (one line per paragraph)
* fix: address cubic review — stale doc facts, make dev/full targets, link checker query strings
- credentials.md: only PROVIDER_CONFIG exists as a map; Vertex/Azure/
OpenAI-compatible provisioning is inline in _provision_*() functions
- content-processing.md: correct ContextConfig priority weights
(source 100 > insight 75 > note 50)
- development-setup.md + Makefile: make dev/full pointed at root compose
files that don't exist; targets now use examples/docker-compose-dev.yml
and examples/docker-compose-full-local.yml with --project-directory .
- check_md_links.py: strip query strings before file-existence checks
* fix: surface silent command-submission failures where they matter
Source.add_insight() caught submission failures and returned None
instead of raising - callers (transformation.py, source.py) run inside
surreal-commands jobs whose outer exception handling already
retries/fails on this, so a swallowed submission failure meant a
transformation could report success while the insight was silently
never persisted. Now raises DatabaseOperationError, matching
vectorize()'s existing contract.
Note.save()'s auto-embed needs the opposite treatment: it's an implicit
side effect of save() (not an explicit dedicated call), and the note
itself is already durably saved by the time it runs - so a submission
hiccup there shouldn't turn an otherwise-successful save into a 500.
Wrapped in try/except, logs and returns None. api/routers/embedding.py's
explicit POST /embed (item_type=note) is the one caller for whom
submission success genuinely is the point of the call, so it separately
checks for a missing command_id and surfaces that as a failure.
* fix: collapse duplicate error logging in add_insight
logger.error + logger.exception logged the same failure twice; a single
logger.exception call carries both the message and the traceback (same
form vectorize() already uses).
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Two small fixes to the podcast episode-listing path, plus a doc note:
- audio_file is only ever set server-side today from a UUID-named
directory under PODCASTS_FOLDER, so this can't currently be tripped -
but the stream/retry/delete endpoints didn't verify the resolved path
actually stayed within PODCASTS_FOLDER before following it. Add
_is_audio_path_contained() as defense in depth against a future code
path (e.g. importing external audio) setting audio_file to something
else.
- Listing episodes called get_job_detail() -> get_command_status() once
per episode, each its own round trip (no connection pooling). Add
PodcastEpisode.get_job_details_for_commands() to batch-fetch status
for every episode's command in one query instead.
Also documents (docs/7-DEVELOPMENT/security.md) that podcast_creator's
configure("templates", {...}) compiles strings as Jinja2 template
source - the same shape as the SSTI vulnerability fixed in
transformation.py (GHSA-f35w-wx37-26q7). Confirmed dormant: no code path
in this repo calls it today. commands/podcast_commands.py gets a
matching code comment warning against wiring user text into it if a
"custom podcast template" feature is ever added.
api/routers/sources.py and api/podcast_service.py interpolated the raw
exception (detail=f"...: {str(e)}") into client-facing error responses -
inconsistent with the safer pattern already used elsewhere in the same
files (e.g. the download handlers), which log the raw exception
server-side but return a fixed generic message. Internal details (DB
hostnames, connection errors, stack-trace fragments) could leak to any
API caller through a 500 response.
Every occurrence already had a matching logger.error() call, so this is
a client-facing message change only, not a logging change. Deliberate
app-authored messages (InvalidInputError text, "Notebook X not found",
result.error_message) are untouched - only raw f"{str(e)}" interpolation
is affected.
tests/test_config_endpoint_no_leak.py is a regression lock for
api/routers/config.py (already correct, not touched by this diff) rather
than a fix - added since it shares the same "unauthenticated endpoint,
don't leak exception text" concern.
* fix: restrict CreateCredentialRequest.provider to a known allowlist
provider was a bare `str`, so any string (typo'd or bogus) flowed
through validation to the domain layer and failed later with a less
clear error, instead of a clean 422 at the API boundary.
Add a SupportedProvider Literal covering the 17 providers already
handled elsewhere - kept in sync with the frontend's ALL_PROVIDERS,
connection_tester.py's TEST_MODELS, and credentials_service.py's
PROVIDER_ENV_CONFIG (a test asserts all three agree on the same set).
* test: lock the frontend provider list into the sync test
The frontend's ALL_PROVIDERS copy was only cross-checked in a docstring;
extract its string literals from the source so all four provider lists
(Literal, TEST_MODELS, PROVIDER_ENV_CONFIG, frontend) are enforced by CI.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Three small, independent hardening fixes to source ingestion:
- generate_unique_filename() checked `if not resolved.exists()` then let
a separate write happen later - two concurrent uploads landing on the
same candidate name could both pass the check and clobber each other.
Now atomically claims the name via Path.touch(exist_ok=False) (O_EXCL)
as part of the search loop itself.
- _resolve_source_file() and _is_source_file_available() compared
`resolved_path.startswith(safe_root)` without a trailing separator - a
sibling directory that merely starts with the same string (e.g.
"uploads_evil/") would incorrectly be treated as contained, unlike this
file's other two path checks which already guard with `+ os.sep`. Not
reachable today (source.asset.file_path is only ever set server-side),
but this closes the gap and matches the existing pattern.
- SourceCreate.notebooks/transformations had no length limit; both are
iterated with a per-item DB lookup in create_source(), so an unbounded
array let a single request trigger an unbounded number of sequential DB
round trips. Capped at 50.
tests/test_upload_type_mitigations.py adds no code change - it documents
why the adjacent "no file type allowlist on uploads" finding was
investigated and judged low-risk without one (downloads are already
served as application/octet-stream regardless of actual file type).
* fix: reject oversized request bodies before auth/routing
No limit existed on request body size, so a single upload could exhaust
memory/disk before any validation ran. MaxBodySizeMiddleware
(api/middleware.py) rejects requests over OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB
(default 100MB) via both a Content-Length pre-check and by counting bytes
as the body streams in, so it also catches chunked requests with no
Content-Length header.
Registered after PasswordAuthMiddleware (wrapping it) so oversized
requests are rejected before spending any work on credential checks, and
inside CORSMiddleware so a rejected upload still gets CORS headers.
* fix: clamp non-positive size limits and log rejected requests
OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB=0 (or negative) made every request with a
body 413; fall back to the default instead, with a warning. Also log each
413 rejection so operators can diagnose failed uploads.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: don't combine wildcard CORS origins with allow_credentials
Combining allow_origins=["*"] with allow_credentials=True makes
Starlette's CORSMiddleware reflect the request's Origin header verbatim
instead of returning a literal "*" (browsers reject a literal wildcard
alongside credentials) - defeating the origin allowlist for any
credentialed request.
allow_credentials is now tied to whether CORS_ORIGINS was explicitly
scoped: False for the default wildcard, True once an operator opts into
specific origins. _cors_headers() (the manual CORS builder for error
responses raised before CORSMiddleware runs) is updated to match, so it
can't grant credentials the real middleware wouldn't.
Not independently exploitable today (the frontend never sends
credentialed requests, and auth is a Bearer header, not a cookie), but
there's no reason to allow it for the default wildcard case.
* fix: key allow_credentials on the parsed origins list, not the env var
An operator who explicitly sets CORS_ORIGINS=* got allow_credentials=True
with a wildcard origin list - the exact reflect-any-Origin behavior this
change exists to prevent. Introduce CORS_ALLOW_CREDENTIALS keyed on the
parsed list containing '*' and use it at both the middleware registration
and the manual error-response headers; replace the tautological formula
tests with ones exercising the real parser.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: don't leak filesystem info via Vertex credential test errors
Vertex's credentials_path is free text with no path validation, and
Google's auth library raises distinguishable exceptions for "file
missing" (OSError), "not valid JSON" (json.JSONDecodeError), and "valid
JSON but wrong shape" (GoogleAuthError). Echoing any of these back to an
API caller turns credential/model testing into a filesystem oracle - an
attacker who can create/test a Vertex credential could probe for the
existence and contents-shape of arbitrary files on the server.
_is_vertex_credentials_file_error() catches all three and both call
sites (api/credentials_service.py's test_credential, connection_tester's
test_individual_model) return one generic message instead.
* fix: don't classify network failures as credentials-file errors
ConnectionError/TimeoutError are OSError subclasses and TransportError is
a GoogleAuthError subclass, so the oracle guard also swallowed genuine
network failures - a user with a blocked network would be told to go debug
their credentials file. Excluding them reveals only the error's category,
which keeps the filesystem oracle closed.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: sanitize raw HTML in the note/transformation markdown preview
MarkdownEditor's live preview renders through @uiw/react-markdown-
preview, which parses literal HTML in the markdown source into real
elements (its `raw` default) - including a live <iframe>. Notes can
hold AI-generated content that echoes an indirect prompt injection
from an ingested document, so this was reachable without the user
writing any HTML themselves.
Add rehype-sanitize (default schema) ahead of rehype-katex in the
preview pipeline. Ordering matters: sanitizing after katex strips
katex's own generated markup (not in the default allowlist), while
sanitizing before it only touches the raw-HTML-derived tree and
leaves not-yet-rendered math nodes alone. Verified against the real
preview component that this strips <iframe>/<script>/<style>/
javascript: URLs while fully preserving math, syntax highlighting,
and GFM tables/task-lists.
* fix: make validate_url() async so DNS resolution doesn't block the event loop
socket.getaddrinfo() in validate_url()'s hostname-resolution branch ran
synchronously on every call - and this is called on the hot path of model
provisioning, potentially once per chat message/transformation. A slow or
hanging DNS lookup stalled every other concurrent request, not just the
one that triggered it.
Run the resolution via asyncio.to_thread() instead, and thread `await`
through every call site: credentials_service.py's discover functions,
routers/credentials.py's create/update, routers/sources.py's source-URL
ingestion, and connection_tester.py's three provider test functions.
save_uploaded_file() did a plain synchronous open()/write() directly
in the async create_source handler, blocking the event loop - and
every other concurrent request - for the duration of a large upload.
Same bug class the recent chat-graph fix (#971) addressed, just not
applied here.
Move the filesystem work (filename resolution + write) into a sync
helper run via asyncio.to_thread(), matching the pattern already used
for execute_command_sync elsewhere in this file. Confirmed the event
loop stays responsive (ticking normally) during a simulated slow
write, and that errors/cleanup still propagate correctly through the
thread.
Building default chat context, the notebook context endpoint, and
podcast generation all looped over every source in a notebook calling
get_context() -> get_insights(), each a separate query that also pays
its own connection setup (no pooling). A notebook with hundreds of
sources meant hundreds of serialized round trips before a chat
message even reached the LLM.
Add SourceInsight.get_for_sources() to fetch insights for every
source in one query, and thread an optional pre-fetched insights list
through Source.get_context() so callers can opt in without changing
its behavior for anyone who doesn't. Measured against a real
(embedded) SurrealDB instance: 14 queries down to 3 for a 12-source
notebook, with correctness verified. The two router call sites treat
a batch-fetch failure the same way the old per-source loop treated a
single failure - falls back to empty insights rather than failing the
whole request.
validate_url() only ran when a credential was created or updated. The
actual HTTP requests (connection testing, model discovery, and real
inference through Esperanto) re-resolve DNS fresh on every call, so a
hostname that resolved to a public IP at save time can later be
repointed to an internal or cloud-metadata address - a classic
DNS-rebinding TOCTOU that a one-time check can't catch.
Move validate_url() to open_notebook/utils so it can be re-run from
the AI layer without an api-depends-on-open_notebook layering
violation, and re-check immediately before every outbound request:
connection_tester's three providers, credential model discovery, and
ModelManager.get_model() on the real-inference path. Also close a
second gap in the same validator: it missed AWS's IPv6 metadata
address (fd00:ec2::254), which isn't link-local so the existing check
never caught it.
POST /sources with type=link copied the user-supplied URL straight
into content-core's fetch with zero validation - unlike the
credential-URL path, which already blocks internal/metadata
addresses. Any user could make the server fetch cloud metadata
endpoints or scan the internal network via "add a web source".
Reuse the same validate_url() guard at the point the URL is first
accepted, before it's ever handed to content-core.
PasswordAuthMiddleware and check_api_password compared the bearer
token to the configured password with `!=`, a timing side-channel on
the single secret gating the whole API. There's no rate limiting to
blunt repeated probing, so switch both to secrets.compare_digest().
repo_relate() interpolated the relate target directly into the query
string, reachable via an unvalidated notebook_id on the save-insight-
as-note flow - a crafted ID could inject and execute arbitrary
SurrealQL (confirmed against a live embedded instance: a single
crafted RELATE call wiped an entire table). Bind record identifiers
as query parameters instead of building them into the query text, and
validate notebook_id exists before relating, matching the pattern
already used by every other caller of add_to_notebook().
* fix: make API startup resilient to a not-yet-ready database
Fixes#708
* fix: bound the database readiness probe with a per-attempt timeout
_wait_for_database awaited migration_manager.ping() with no timeout, so a
hung connection probe could bypass the retry budget and block startup
indefinitely. Wrap the probe in asyncio.wait_for with a per-attempt ceiling;
a timed-out probe is treated as a transient failure and retried like any
other unreachable-database attempt, keeping the retry budget bounded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for startup database readiness retry (#708)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* feat: Recently Viewed section for notebooks and sources
Fixes#850
* fix: export RecentlyViewedResponse type from api types
The RecentlyViewed component and notebooks API client import
RecentlyViewedResponse from @/lib/types/api but the interface was missing,
failing the type-check build. Add it to match the backend api/models.py model.
* fix: harden recently-viewed read path and index the recency query
Three robustness fixes on the recently-viewed feature:
- Make the last_viewed_at write-on-read stamping best-effort. _stamp_source_view
and _stamp_notebook_view now swallow and log their own errors, so a failed
stamp update can no longer turn a successful GET /sources/{id} or
GET /notebooks/{id} into a 500.
- Stop leaking internal details from GET /recently-viewed: log the full
exception server-side and return a generic error message to the client.
- Add indexes on last_viewed_at for the notebook and source tables in
migration 16 so the ORDER BY last_viewed_at DESC LIMIT recently-viewed
query does not degrade into a full table scan as data grows. The down
migration removes the indexes before dropping the field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: register migration 18, add recently-viewed i18n keys and CHANGELOG entry
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* feat: per-transformation custom model selection
Fixes#776
* fix: validate model_id exists when creating or updating a transformation
The create and update endpoints persisted model_id without checking the
referenced model exists, so an invalid reference was stored silently and only
surfaced later as a 404 at execution time. Add the same existence check
execute_transformation already performs (Model.get -> 404 if missing) to both
the create and update paths, so a bad model_id is rejected up front with a
clear 404. Update still allows clearing model_id to None.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: cover model_id existence validation on transformation create/update
The create test now mocks Model.get so the new existence check passes and
asserts it was awaited with the supplied model_id. The update test expects
Model.get to be awaited twice (once validating the update, once at execute)
rather than once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for per-transformation model selection (#776)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: pass max_tokens through to podcast_creator for outline/transcript generation
* fix: persist and expose episode_profile max_tokens (migration + API)
* docs: add CHANGELOG entry for episode profile max_tokens pass-through (#639)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Add type annotations to PasswordAuthMiddleware.__init__ and dispatch (plus
excluded_paths) and drop the [mypy-api.auth] ignore_errors stanza so the
module is type-checked. Annotations only; no behavior change.
Refs #945
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Expand /api/sources sorting to type, title, created, updated, insights_count, and embedded (whitelisted, with a computed type alias so SurrealDB can order by it). Frontend adds sortable headers for every column plus an Updated column.
Fixes#895
Notebook chat (execute_chat) and source-chat SSE ran LangGraph's synchronous invoke() on the event loop, freezing the whole API while the LLM responded. Both now run via asyncio.to_thread(), matching the existing get_state() calls.
Fixes#704
Switches the streaming endpoints (/api/search/ask, source chat messages) to media_type=text/event-stream and adds dedicated App Router SSE route handlers (_sse-proxy.ts) that stream the upstream body directly, bypassing the Next.js rewrites proxy's gzip buffering in standalone/production mode.
Verified locally: with Accept-Encoding: gzip (i.e. a real browser), the standalone rewrites proxy buffered the entire SSE stream to completion; the route-handler approach restores progressive delivery.
* fix(api): return 404 instead of 500 for missing resources in CRUD endpoints (#862)
ObjectModel.get() raises NotFoundError for a missing record (never returns a
falsy value), so the 'if not obj: 404' guards were dead code and each handler's
broad 'except Exception' re-raised NotFoundError as 500, never hitting the global
NotFoundError->404 handler.
Add an explicit 'except NotFoundError -> 404' arm to the affected handlers in
notebooks, notes, models, credentials and embedding routers, plus regression
tests asserting 404 (not 500) when .get() raises NotFoundError.
* refactor(api): remove dead 'if not obj' guards after .get() (cubic #924)
Now that each handler maps NotFoundError -> 404, the 'if not obj: raise 404'
guards are unreachable (.get() raises rather than returning a falsy value).
Drop them; where the fetched object was only used by the guard, keep the .get()
call for its existence-validation side effect without binding an unused var.
* chore(release): prepare v1.10.0
* style: fix import ordering flagged by ruff
* fix(sources): return 404 for missing source and fix retry 500 from double-prefixed command id
- GET /sources/{id} mapped NotFoundError to a generic 500; now returns 404
- POST /sources/{id}/retry double-prefixed the command id (command:command:...),
raising 'too many values to unpack' after queuing; align with the create path
- update retry test mock to realistic prefixed command id + guard against
double-prefix; add 404 regression test
* fix(sources): mark failed extraction as failed so retry surfaces (#726)
content-core signals soft failures (unreachable/invalid URL) by returning
title=Error + 'Failed to extract content:' body instead of raising, and the
process_source command swallowed permanent ValueErrors into a success=False
result. Since surreal-commands marks a job completed when the function returns,
failed ingests showed status 'completed' and never offered the retry button.
- source graph: detect the content-core failure sentinel and raise
- process_source_command: re-raise ValueError (stop_on already prevents retry)
so the job is marked failed and the source becomes retryable
* feat(notebook): per-type bulk context actions for sources and notes (#223)
- Sources context menu now offers 'insights only' (sources without insights
are excluded rather than forced to full), 'full content', and 'exclude all'
- Add the same bulk Context menu to the Notes column (include all / exclude all)
- Bulk choices propagate to items loaded later via pagination
- New locale keys (includeAllInsights/includeAllFull) across all 14 locales
- Unit tests for the new bulk modes and note context helpers
* fix(search): reject non-positive limit and survive highlight overflow
- SearchRequest.limit now has ge=1 so 0/negative values are rejected with
422 instead of flowing into SurrealDB as LIMIT -1/0 (500 or empty set) (#863)
- text_search() catches SurrealDB's "position overflow" highlight error and
falls back to vector_search() so large/multi-byte chunks no longer 500 (#648)
- add tests covering limit validation and the overflow fallback path
* fix(search): surface error when vector fallback also fails
Address cubic review on #898:
- text_search() no longer returns [] when the vector-search fallback also
fails; it logs the traceback and raises DatabaseOperationError so a total
search outage is not masked as a legitimate empty result set
- document the fallback + failure behavior in domain/CLAUDE.md
- update the test to assert the double-failure path raises
POST /sources/{id}/retry looked up a source's notebooks with
`SELECT notebook FROM reference WHERE source = $source_id`, but `reference`
is a graph edge (RELATE source->reference->notebook) with only `in`/`out`
columns. The query matched nothing, so `notebook_ids` was always empty and
the endpoint returned 400 for every source. Mirror the working query used in
the source-list path: `SELECT VALUE out FROM reference WHERE in = $source_id`.
Adds regression tests asserting retry re-queues a linked source and only 400s
when a source is genuinely unlinked.
PUT /api/episode-profiles/{id} and PUT /api/speaker-profiles/{id}
both blindly assigned every field of the request body model (including
Optional fields with None defaults) onto the DB object. SurrealDB
rejected None values for fields typed as string — causing HTTP 500
whenever the client omitted any optional field.
Replace the per-field assignment blocks with a model_dump(exclude_unset=True)
loop so only fields the client actually included are written through.
Fields absent from the request body retain their current DB values.
Closes#809
Co-authored-by: james <li@jamesdeMacBook-Pro.local>
OpenRouter embeddings work end-to-end as of 1.9.0 (Esperanto 2.22 fixed the
malformed embedding request body, and the frontend already offers the
embedding modality for OpenRouter). Align the backend so it's consistent:
- credentials_service PROVIDER_MODALITIES: openrouter -> [language, embedding]
so env/legacy credential migration also enables embedding for OpenRouter
(get_default_modalities).
- README provider matrix: OpenRouter embedding ✅.
Closes#717
Enable the remaining esperanto audio modalities for providers Open Notebook
already supports:
- google: + speech_to_text, text_to_speech
- vertex: + text_to_speech
- elevenlabs: + speech_to_text (Scribe)
classify_model_type gains a Google TTS pattern ("tts", matched before the
broad gemini language pattern) and an ElevenLabs STT pattern ("scribe", matched
before the "eleven" TTS pattern). Google STT reuses plain Gemini names that are
indistinguishable from language models by name, so it has no pattern — users
assign the speech_to_text type manually when registering. ElevenLabs discovery
now also returns scribe_v1 as a speech_to_text model.
The frontend already exposed these modalities; this aligns the backend
PROVIDER_MODALITIES, classification, and discovery with esperanto and the UI.
Closes#828
Surface the audio providers added in esperanto 2.21/2.22:
- Mistral Voxtral STT (voxtral-*-latest) and TTS (voxtral-mini-tts) — reuses
MISTRAL_API_KEY; discovered via the existing /v1/models endpoint.
- Deepgram TTS (Aura) — new provider: DEEPGRAM_API_KEY env config, key_provider
mapping, static Aura voice catalog for discovery, modality + display name +
docs link in the UI.
- xAI TTS — adds text_to_speech to the existing xai provider. xAI TTS is
voice-based and sends no model id, so the model name is cosmetic; users add it
via the custom-model input (no discovery entry).
classify_model_type now distinguishes Voxtral TTS vs STT (the "-tts" model must
not be caught by the broader STT names, since STT is checked first) and Aura
voices. PROVIDER_MODALITIES, PROVIDER_ENV_CONFIG, discover_with_config static
list, TEST_MODELS, DEFAULT_TEST_VOICES, the provider-availability endpoint, and
the frontend provider constants are all updated. Provider display names are
frontend constants (not i18n), so no locale changes are needed.
Also removes the dead test_provider_connection() function (no callers; the UI
uses test_credential/test_individual_model) and its now-unused imports.
Closes#826Closes#827
Upgrade esperanto 2.20.0 -> 2.22.0. The constraint (>=2.20.0,<3) already
allowed it; this relocks and picks up upstream fixes (OpenRouter json body,
clearer null-embedding errors, streaming ToolCall objects, base_url
trailing-slash normalization, Ollama thinking-model content).
Esperanto 2.21.0 lowered the Ollama num_ctx default from 128000 to 8192 to
avoid OOM on consumer GPUs. We keep that safe default and add an optional
per-credential num_ctx override for self-hosters whose hardware can handle a
larger context window:
- Credential gains a num_ctx field, surfaced via to_esperanto_config() so it
flows into AIFactory automatically (no ModelManager change needed).
- Credential create/update API schemas + router pass num_ctx through.
- Frontend: optional numeric field on the Ollama credential form, with i18n
labels translated across all 13 locales.
- Docs: document the new default and the override under AI providers.
Closes#825
* Fix Ollama model discovery and local Docker config
* fix: revert unrelated docker-compose changes, keep ollama embedding classification
Restore docker-compose.yml to upstream (local dev build/port changes were
out of scope), and fix PEP8 blank line in credentials_service.py.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Fixes#676 — OpenAI model discovery always called https://api.openai.com/v1/models,
ignoring a credential's configured base_url. Use the configured base_url for
the openai provider and add a models_endpoint() helper that avoids appending
/models twice when the endpoint already includes it.
Fixes#792 — credentials for the openai_compatible provider could not be
tested or used because api/routers/models.py spelled the provider with a
hyphen while the rest of the codebase (allow-list, credentials_service,
frontend) uses the underscore form. Align the models endpoint to the
underscore form, normalizing to the hyphen only at the Esperanto boundary
where AIFactory reports it with a hyphen.
The chat system prompt template had a conditional block for notebook
metadata but the /chat/execute endpoint never populated it. Fetch the
notebook linked to the session via the refers_to relationship and pass
it to the graph state so the LLM receives the notebook's name and
description as project context.
Closes#685
Replace hardcoded `allow_origins=["*"]` with a parsed `CORS_ORIGINS`
environment variable (comma-separated). Default remains `*` for
backward compatibility — no existing deployment breaks — but the API
now logs a startup warning prompting users to set it explicitly for
production.
Exception handlers now route their CORS headers through a shared
`_cors_headers()` helper that mirrors Starlette's CORSMiddleware
behavior: reflects the request Origin when allowed (handling the
browser-rejected `*` + credentials combination correctly), and omits
`Access-Control-Allow-Origin` for disallowed origins so error bodies
don't leak cross-origin when `CORS_ORIGINS` is configured.
Closes#585, #730.
Based on the original work by Greg Grace in #597; rewritten on top of
current main to address prior review feedback (load_dotenv kept at
top, `import os` grouped with stdlib, `_cors_headers` defined before
its exception-handler callers, origins parsed once at module load)
and to choose a non-breaking default paired with a startup warning
instead of a stricter-by-default origin.
Co-authored-by: Greg Grace <ggrace@519lab.com>
- Catch only ValueError (decryption errors) instead of broad Exception
so NotFoundError and other failures propagate correctly
- Support migrate_to parameter in the fallback delete path so linked
models can be reassigned instead of always cascade-deleted
- Sanitize decryption_error message to not expose raw exception details
- Credential.get_all() now uses per-row error handling instead of failing on first bad row
- Broken credentials include decryption_error field with descriptive message
- DELETE endpoint falls back to direct DB delete when credential can't be decrypted
- Frontend shows amber warning alert for broken credentials with disabled test/edit/discover
- Added i18n translation keys for decryption error warning in all 9 locales
Append os.sep to the directory path before startswith() check so that
paths like /app/data/uploads_evil/ cannot bypass the uploads directory
validation.
- Bump ai-prompter to >=0.4.0 which uses Jinja2 SandboxedEnvironment,
preventing arbitrary code execution via user-provided transformation prompts
- Sanitize uploaded filenames with os.path.basename() and validate resolved
path stays within upload directory to prevent path traversal
- Validate file_path in source creation is within UPLOADS_FOLDER to prevent
arbitrary file read via Local File Inclusion
- Bump esperanto dependency to >=2.20.0 for new provider profiles
- Register both providers in credentials, key provider, connection tester, model discovery, and models router
- Add frontend provider entries (display names, modalities, docs links)
- Add documentation sections for both providers in ai-providers.md, environment-reference.md, and provider comparison
- #627: Set source.asset (URL/file_path) before save() in async creation
path so failed sources are identifiable and retry works
- #670: Only overwrite source title if it's a placeholder ("Processing...")
or empty, preserving user-set custom titles
- #651: Cascade-delete linked models when credential is deleted instead of
returning 409 Conflict; remove unused delete_models parameter
- Add tests for all three fixes (12 new tests)
- Add .harness and .mcp.json to .gitignore
