* fix: make the provider connection test resilient to model retirement
Two independent hardenings against the #970 class of breakage (a hard-coded
Gemini test model getting shut down by Google, which made testing a valid
key fail with a 404):
- Use Google's floating alias gemini-flash-latest for the Google/Vertex
test model instead of a dated id, so a retirement repoints it for us.
- Reframe the provider connection test around what an error actually
proves: only a rejected key (401), missing permissions (403), or an
unreachable endpoint are failures. Anything the provider returns after
authenticating - a rate limit, or a missing/retired/unsupported model -
still proves the credentials work, so it reports success. Previously this
relied on matching the literal phrase 'not found' + 'model', which a
differently-worded retirement/deprecation error slipped past.
Unifies the auth/network/rate-limit classification (previously duplicated
and divergent between connection_tester and credentials_service) into
shared helpers. The individual-model test keeps model-not-found as a
failure, since there a specific registered model really is broken.
Adds classification tests with realistic provider error strings.
* docs: add CHANGELOG entry for connection-test resilience fix
* fix: update Google model version in connection tester and tests
* fix: updated the gemini model lists that references deprecated models
* fix: update also documentation. This is more prone to not follow the maintainer's directives, so PTADL
* fix: forgot two references to gemini deprecated versions
* fix: use valid, longer-lived gemini model IDs
Several IDs the PR introduced don't exist or are near shutdown, verified
against Google's official model/deprecation pages (2026-07):
- connection test model gemini-2.5-flash -> gemini-3.5-flash (2.5-flash
retires 2026-10-16; 3.5-flash is the current stable GA and Google's
named replacement, so the #970 fix doesn't re-break in ~3 months)
- gemini-3.5-pro (does not exist) -> gemini-2.5-pro in docs
- plain gemini-3.1-flash (not a GA Gemini-API id) dropped from the Vertex
discovery list / preferences / docs; use gemini-3.5-flash or -flash-lite
- dead gemini-pro dropped from preferences and docs
Ported #996's #970 regression test, pinned to gemini-3.5-flash.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* docs: restructure documentation around AGENTS.md, VISION.md and decision records
- Consolidate 17 CLAUDE.md files into 3 AGENTS.md (root, backend, frontend);
CLAUDE.md files become @AGENTS.md pointers
- Add VISION.md: product identity + current posture with horizon clusters
- Add docs/7-DEVELOPMENT/decisions/ with 4 retroactive ADRs and 2 PDRs
- Add 5 new engineering docs pages (credentials, content-processing,
podcasts, prompts, frontend) absorbing knowledge from removed CLAUDE.md
- Dismember TRIAGE.md: label taxonomy into maintainer-guide.md, product
jurisprudence into VISION.md, operator heuristics stay local (gitignored)
- Add AI-assisted/agent-generated PR guidelines to contributing.md
- Convert README.dev.md into a pointer after migrating its unique content
(make workflow matrix, Docker publishing, add-a-language playbook)
- Fix stale docs: migration path/format, provider count, locale list;
fix broken links (docs/index.md, PR template, CONFIGURATION.md)
* docs: fix README doc links and add markdown link check to CI
- Repoint 9 README links to pages that actually exist in docs/
- Replace literal (link) placeholder in maintainer-guide templates
- Add scripts/check_md_links.py validating relative links in tracked
markdown (skips URLs, anchors and code spans)
- Add docs-links workflow running the check on PRs that touch markdown
* docs: add documentation restructure to changelog
* feat: add cubic.yaml with project-aware AI review agents
Three custom review agents (vision & principles alignment backed by
VISION.md, known mechanical caveats, security & testability), PR-contract
review instructions, and automatic ultrareviews for auth, credential,
encryption and migration changes.
* docs: graduate issue-first policy by change size
Small obvious fixes (typos, docs, tiny bugs, i18n completions) no longer
require an issue; features and architecture changes still do. Sizeable
PRs opened without an issue convert to draft while the issue goes
through triage (1-2 days). Applied consistently across contributing
guide, root CONTRIBUTING pointer, PR template, maintainer guide red
flags and cubic review instructions.
* docs: align PR template Related Issue section with graduated issue-first policy
* docs: address review — generalize ADR-002/004, unwrap hard-wrapped lines
- ADR-002 now records the general delegation rule (platform/media support
that needs heavy coding lives in focused external libraries) covering
Esperanto, Content Core and podcast-creator
- ADR-004 now records the durable decision (long-running work runs on
background workers — heavy content, varied machine sizes, never lock
usage) with the queue technology as a swappable implementation detail
pending #381
- Remove mid-paragraph hard line wrapping from authored docs to match
repo convention (one line per paragraph)
* fix: address cubic review — stale doc facts, make dev/full targets, link checker query strings
- credentials.md: only PROVIDER_CONFIG exists as a map; Vertex/Azure/
OpenAI-compatible provisioning is inline in _provision_*() functions
- content-processing.md: correct ContextConfig priority weights
(source 100 > insight 75 > note 50)
- development-setup.md + Makefile: make dev/full pointed at root compose
files that don't exist; targets now use examples/docker-compose-dev.yml
and examples/docker-compose-full-local.yml with --project-directory .
- check_md_links.py: strip query strings before file-existence checks
* fix: prevent SurrealQL injection in Credential.get_all()'s order_by
ObjectModel.get_all() already validates order_by against an allowlist
before interpolating it into the query string - but Credential overrides
get_all() (to handle per-row api_key decryption) and its override built
the ORDER BY clause with a raw, unvalidated f-string, bypassing that
protection entirely.
Extract the base class's validation logic into a reusable
_validate_order_by() classmethod and route Credential.get_all() through
it. Not reachable from any current API surface today (the only caller
passes a hardcoded "provider, created", not user input), but any
subclass that builds its own query around order_by instead of
delegating to the base get_all() needs this so the allowlist can't
silently drift between call sites.
No test currently exercises this path - worth adding one (e.g.
Credential.get_all(order_by="field; DROP TABLE credential") raises
InvalidInputError) before merging.
* test: add order_by injection regression tests
Covers _validate_order_by() normalization/rejection and the Credential
.get_all() path, which raises on an injected clause before any query
reaches the database.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: surface silent command-submission failures where they matter
Source.add_insight() caught submission failures and returned None
instead of raising - callers (transformation.py, source.py) run inside
surreal-commands jobs whose outer exception handling already
retries/fails on this, so a swallowed submission failure meant a
transformation could report success while the insight was silently
never persisted. Now raises DatabaseOperationError, matching
vectorize()'s existing contract.
Note.save()'s auto-embed needs the opposite treatment: it's an implicit
side effect of save() (not an explicit dedicated call), and the note
itself is already durably saved by the time it runs - so a submission
hiccup there shouldn't turn an otherwise-successful save into a 500.
Wrapped in try/except, logs and returns None. api/routers/embedding.py's
explicit POST /embed (item_type=note) is the one caller for whom
submission success genuinely is the point of the call, so it separately
checks for a missing command_id and surfaces that as a failure.
* fix: collapse duplicate error logging in add_insight
logger.error + logger.exception logged the same failure twice; a single
logger.exception call carries both the message and the traceback (same
form vectorize() already uses).
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Two small fixes to the podcast episode-listing path, plus a doc note:
- audio_file is only ever set server-side today from a UUID-named
directory under PODCASTS_FOLDER, so this can't currently be tripped -
but the stream/retry/delete endpoints didn't verify the resolved path
actually stayed within PODCASTS_FOLDER before following it. Add
_is_audio_path_contained() as defense in depth against a future code
path (e.g. importing external audio) setting audio_file to something
else.
- Listing episodes called get_job_detail() -> get_command_status() once
per episode, each its own round trip (no connection pooling). Add
PodcastEpisode.get_job_details_for_commands() to batch-fetch status
for every episode's command in one query instead.
Also documents (docs/7-DEVELOPMENT/security.md) that podcast_creator's
configure("templates", {...}) compiles strings as Jinja2 template
source - the same shape as the SSTI vulnerability fixed in
transformation.py (GHSA-f35w-wx37-26q7). Confirmed dormant: no code path
in this repo calls it today. commands/podcast_commands.py gets a
matching code comment warning against wiring user text into it if a
"custom podcast template" feature is ever added.
* fix: don't leak filesystem info via Vertex credential test errors
Vertex's credentials_path is free text with no path validation, and
Google's auth library raises distinguishable exceptions for "file
missing" (OSError), "not valid JSON" (json.JSONDecodeError), and "valid
JSON but wrong shape" (GoogleAuthError). Echoing any of these back to an
API caller turns credential/model testing into a filesystem oracle - an
attacker who can create/test a Vertex credential could probe for the
existence and contents-shape of arbitrary files on the server.
_is_vertex_credentials_file_error() catches all three and both call
sites (api/credentials_service.py's test_credential, connection_tester's
test_individual_model) return one generic message instead.
* fix: don't classify network failures as credentials-file errors
ConnectionError/TimeoutError are OSError subclasses and TransportError is
a GoogleAuthError subclass, so the oracle guard also swallowed genuine
network failures - a user with a blocked network would be told to go debug
their credentials file. Excluding them reveals only the error's category,
which keeps the filesystem oracle closed.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: sanitize raw HTML in the note/transformation markdown preview
MarkdownEditor's live preview renders through @uiw/react-markdown-
preview, which parses literal HTML in the markdown source into real
elements (its `raw` default) - including a live <iframe>. Notes can
hold AI-generated content that echoes an indirect prompt injection
from an ingested document, so this was reachable without the user
writing any HTML themselves.
Add rehype-sanitize (default schema) ahead of rehype-katex in the
preview pipeline. Ordering matters: sanitizing after katex strips
katex's own generated markup (not in the default allowlist), while
sanitizing before it only touches the raw-HTML-derived tree and
leaves not-yet-rendered math nodes alone. Verified against the real
preview component that this strips <iframe>/<script>/<style>/
javascript: URLs while fully preserving math, syntax highlighting,
and GFM tables/task-lists.
* fix: make validate_url() async so DNS resolution doesn't block the event loop
socket.getaddrinfo() in validate_url()'s hostname-resolution branch ran
synchronously on every call - and this is called on the hot path of model
provisioning, potentially once per chat message/transformation. A slow or
hanging DNS lookup stalled every other concurrent request, not just the
one that triggered it.
Run the resolution via asyncio.to_thread() instead, and thread `await`
through every call site: credentials_service.py's discover functions,
routers/credentials.py's create/update, routers/sources.py's source-URL
ingestion, and connection_tester.py's three provider test functions.
Building default chat context, the notebook context endpoint, and
podcast generation all looped over every source in a notebook calling
get_context() -> get_insights(), each a separate query that also pays
its own connection setup (no pooling). A notebook with hundreds of
sources meant hundreds of serialized round trips before a chat
message even reached the LLM.
Add SourceInsight.get_for_sources() to fetch insights for every
source in one query, and thread an optional pre-fetched insights list
through Source.get_context() so callers can opt in without changing
its behavior for anyone who doesn't. Measured against a real
(embedded) SurrealDB instance: 14 queries down to 3 for a 12-source
notebook, with correctness verified. The two router call sites treat
a batch-fetch failure the same way the old per-source loop treated a
single failure - falls back to empty insights rather than failing the
whole request.
validate_url() only ran when a credential was created or updated. The
actual HTTP requests (connection testing, model discovery, and real
inference through Esperanto) re-resolve DNS fresh on every call, so a
hostname that resolved to a public IP at save time can later be
repointed to an internal or cloud-metadata address - a classic
DNS-rebinding TOCTOU that a one-time check can't catch.
Move validate_url() to open_notebook/utils so it can be re-run from
the AI layer without an api-depends-on-open_notebook layering
violation, and re-check immediately before every outbound request:
connection_tester's three providers, credential model discovery, and
ModelManager.get_model() on the real-inference path. Also close a
second gap in the same validator: it missed AWS's IPv6 metadata
address (fd00:ec2::254), which isn't link-local so the existing check
never caught it.
transformation.prompt and the generic pattern-chain's prompt were
passed to Prompter(template_text=...), compiling attacker-influenced
text directly as Jinja2 template source. ai-prompter's sandboxed
environment blocks the classic __globals__/__subclasses__ RCE
gadgets, but not an unbounded {% for %} loop - a trivial DoS for any
authenticated user, and one instance of the same "user text becomes
template source" pattern behind a previously-disclosed critical CVE
(GHSA-f35w-wx37-26q7), whose fix only added sandboxing without
removing the pattern itself.
Render through fixed, developer-authored templates instead, with the
user's text passed in as a plain variable. Verified byte-identical
output for legitimate prompts and confirmed the same payload that
used to be a DoS vector now renders as inert text in under a
millisecond.
repo_relate() interpolated the relate target directly into the query
string, reachable via an unvalidated notebook_id on the save-insight-
as-note flow - a crafted ID could inject and execute arbitrary
SurrealQL (confirmed against a live embedded instance: a single
crafted RELATE call wiped an entire table). Bind record identifiers
as query parameters instead of building them into the query text, and
validate notebook_id exists before relating, matching the pattern
already used by every other caller of add_to_notebook().
* fix: make API startup resilient to a not-yet-ready database
Fixes#708
* fix: bound the database readiness probe with a per-attempt timeout
_wait_for_database awaited migration_manager.ping() with no timeout, so a
hung connection probe could bypass the retry budget and block startup
indefinitely. Wrap the probe in asyncio.wait_for with a per-attempt ceiling;
a timed-out probe is treated as a transient failure and retried like any
other unreachable-database attempt, keeping the retry budget bounded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for startup database readiness retry (#708)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* feat: Recently Viewed section for notebooks and sources
Fixes#850
* fix: export RecentlyViewedResponse type from api types
The RecentlyViewed component and notebooks API client import
RecentlyViewedResponse from @/lib/types/api but the interface was missing,
failing the type-check build. Add it to match the backend api/models.py model.
* fix: harden recently-viewed read path and index the recency query
Three robustness fixes on the recently-viewed feature:
- Make the last_viewed_at write-on-read stamping best-effort. _stamp_source_view
and _stamp_notebook_view now swallow and log their own errors, so a failed
stamp update can no longer turn a successful GET /sources/{id} or
GET /notebooks/{id} into a 500.
- Stop leaking internal details from GET /recently-viewed: log the full
exception server-side and return a generic error message to the client.
- Add indexes on last_viewed_at for the notebook and source tables in
migration 16 so the ORDER BY last_viewed_at DESC LIMIT recently-viewed
query does not degrade into a full table scan as data grows. The down
migration removes the indexes before dropping the field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: register migration 18, add recently-viewed i18n keys and CHANGELOG entry
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* feat: per-transformation custom model selection
Fixes#776
* fix: validate model_id exists when creating or updating a transformation
The create and update endpoints persisted model_id without checking the
referenced model exists, so an invalid reference was stored silently and only
surfaced later as a 404 at execution time. Add the same existence check
execute_transformation already performs (Model.get -> 404 if missing) to both
the create and update paths, so a bad model_id is rejected up front with a
clear 404. Update still allows clearing model_id to None.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: cover model_id existence validation on transformation create/update
The create test now mocks Model.get so the new existence check passes and
asserts it was awaited with the supplied model_id. The update test expects
Model.get to be awaited twice (once validating the update, once at execute)
rather than once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add CHANGELOG entry for per-transformation model selection (#776)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: pass max_tokens through to podcast_creator for outline/transcript generation
* fix: persist and expose episode_profile max_tokens (migration + API)
* docs: add CHANGELOG entry for episode profile max_tokens pass-through (#639)
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Remove the mypy ignore_errors override for open_notebook.domain.base and fix the surfaced type errors by adding typed DB config helpers (namespace/database default to open_notebook, password to root — matching the documented defaults). Adds tests/test_repository_config.py.
One slice of the incremental mypy cleanup. Refs #945
token_count() now encodes with disallowed_special=() so source/context text containing sequences like <|endoftext|> no longer raises ValueError.
Fixes#667
* chore(release): prepare v1.10.0
* style: fix import ordering flagged by ruff
* fix(sources): return 404 for missing source and fix retry 500 from double-prefixed command id
- GET /sources/{id} mapped NotFoundError to a generic 500; now returns 404
- POST /sources/{id}/retry double-prefixed the command id (command:command:...),
raising 'too many values to unpack' after queuing; align with the create path
- update retry test mock to realistic prefixed command id + guard against
double-prefix; add 404 regression test
* fix(sources): mark failed extraction as failed so retry surfaces (#726)
content-core signals soft failures (unreachable/invalid URL) by returning
title=Error + 'Failed to extract content:' body instead of raising, and the
process_source command swallowed permanent ValueErrors into a success=False
result. Since surreal-commands marks a job completed when the function returns,
failed ingests showed status 'completed' and never offered the retry button.
- source graph: detect the content-core failure sentinel and raise
- process_source_command: re-raise ValueError (stop_on already prevents retry)
so the job is marked failed and the source becomes retryable
* feat(notebook): per-type bulk context actions for sources and notes (#223)
- Sources context menu now offers 'insights only' (sources without insights
are excluded rather than forced to full), 'full content', and 'exclude all'
- Add the same bulk Context menu to the Notes column (include all / exclude all)
- Bulk choices propagate to items loaded later via pagination
- New locale keys (includeAllInsights/includeAllFull) across all 14 locales
- Unit tests for the new bulk modes and note context helpers
* fix(credentials): persist Ollama num_ctx via a flexible config object
The `credential` table is SCHEMAFULL, so the model's `num_ctx` field was
silently dropped on write and the override never took effect (#875).
Rather than add a typed column per provider option, add a single flexible
`config` object to the credential table (migration 15). Provider-specific
tuning options (currently `num_ctx`) are still exposed as top-level fields on
the Credential model and on the API, but are packed into `config` on save and
lifted back out on load via a before-validator. Future options only need a
Pydantic field + an entry in CONFIG_EXTRAS — no further migrations.
- migration 15 (+ down): DEFINE FIELD config ON credential FLEXIBLE TYPE option<object>
- Credential: CONFIG_EXTRAS set, _lift_config before-validator, config packing
in _prepare_save_data
- tests covering pack/unpack round-trip and the empty-config case
* fix(credentials): preserve unmapped config keys on save
Address cubic review on #903:
- `config` is now a real model field mirroring the credential table's FLEXIBLE
object and is the on-disk source of truth. `num_ctx` remains a convenience
field mirrored from/to config. On save we start from the existing bag and
sync the convenience fields in, so a save never clobbers config keys written
by a newer version (repo_update uses MERGE, which replaces the whole object).
config is only written as None when the merged result is genuinely empty.
(The previous approach relied on Pydantic extras, which ObjectModel's default
extra="ignore" silently dropped.)
- Update database/CLAUDE.md migration totals (15 up/down, incl. migration 15).
- Add tests for unmapped-key preservation and clearing num_ctx while keeping
other config keys.
* fix(credentials): validate config-mirrored extras via Pydantic
Address cubic re-review on #903: mirror known config keys (num_ctx) onto their
convenience fields in a `before` model validator so they go through normal
Pydantic field validation/coercion, instead of an `after`-validator
object.__setattr__ that bypassed type checks. Adds a test asserting num_ctx is
coerced to int and a non-coercible value is rejected.
* fix(search): reject non-positive limit and survive highlight overflow
- SearchRequest.limit now has ge=1 so 0/negative values are rejected with
422 instead of flowing into SurrealDB as LIMIT -1/0 (500 or empty set) (#863)
- text_search() catches SurrealDB's "position overflow" highlight error and
falls back to vector_search() so large/multi-byte chunks no longer 500 (#648)
- add tests covering limit validation and the overflow fallback path
* fix(search): surface error when vector fallback also fails
Address cubic review on #898:
- text_search() no longer returns [] when the vector-search fallback also
fails; it logs the traceback and raises DatabaseOperationError so a total
search outage is not masked as a legitimate empty result set
- document the fallback + failure behavior in domain/CLAUDE.md
- update the test to assert the double-failure path raises
PodcastService expected Notebook.get_context() but it did not exist,
so podcasts were generated from empty/placeholder context. Adds the
method plus opt-in include_full_text / include_content flags on
get_sources()/get_notes() (defaults preserve the existing omit
behavior) and assembles formatted source/note context blocks.
Fixes#808
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The model connection test transcribed a 0.5s silent WAV, so every
speech-to-text test returned a blank transcription — looking broken even
when the provider worked. Bundle a tiny "Hello there" MP3 (6 KB, 16 kHz
mono) and use it instead, so a passing STT test returns actual text.
- Add open_notebook/ai/assets/test_speech.mp3 (shipped via package-data).
- _get_test_audio() loads it, falling back to the silent WAV if missing.
- Empty transcriptions now report a clear message instead of a blank line.
Enable the remaining esperanto audio modalities for providers Open Notebook
already supports:
- google: + speech_to_text, text_to_speech
- vertex: + text_to_speech
- elevenlabs: + speech_to_text (Scribe)
classify_model_type gains a Google TTS pattern ("tts", matched before the
broad gemini language pattern) and an ElevenLabs STT pattern ("scribe", matched
before the "eleven" TTS pattern). Google STT reuses plain Gemini names that are
indistinguishable from language models by name, so it has no pattern — users
assign the speech_to_text type manually when registering. ElevenLabs discovery
now also returns scribe_v1 as a speech_to_text model.
The frontend already exposed these modalities; this aligns the backend
PROVIDER_MODALITIES, classification, and discovery with esperanto and the UI.
Closes#828
Surface the audio providers added in esperanto 2.21/2.22:
- Mistral Voxtral STT (voxtral-*-latest) and TTS (voxtral-mini-tts) — reuses
MISTRAL_API_KEY; discovered via the existing /v1/models endpoint.
- Deepgram TTS (Aura) — new provider: DEEPGRAM_API_KEY env config, key_provider
mapping, static Aura voice catalog for discovery, modality + display name +
docs link in the UI.
- xAI TTS — adds text_to_speech to the existing xai provider. xAI TTS is
voice-based and sends no model id, so the model name is cosmetic; users add it
via the custom-model input (no discovery entry).
classify_model_type now distinguishes Voxtral TTS vs STT (the "-tts" model must
not be caught by the broader STT names, since STT is checked first) and Aura
voices. PROVIDER_MODALITIES, PROVIDER_ENV_CONFIG, discover_with_config static
list, TEST_MODELS, DEFAULT_TEST_VOICES, the provider-availability endpoint, and
the frontend provider constants are all updated. Provider display names are
frontend constants (not i18n), so no locale changes are needed.
Also removes the dead test_provider_connection() function (no callers; the UI
uses test_credential/test_individual_model) and its now-unused imports.
Closes#826Closes#827
Upgrade esperanto 2.20.0 -> 2.22.0. The constraint (>=2.20.0,<3) already
allowed it; this relocks and picks up upstream fixes (OpenRouter json body,
clearer null-embedding errors, streaming ToolCall objects, base_url
trailing-slash normalization, Ollama thinking-model content).
Esperanto 2.21.0 lowered the Ollama num_ctx default from 128000 to 8192 to
avoid OOM on consumer GPUs. We keep that safe default and add an optional
per-credential num_ctx override for self-hosters whose hardware can handle a
larger context window:
- Credential gains a num_ctx field, surfaced via to_esperanto_config() so it
flows into AIFactory automatically (no ModelManager change needed).
- Credential create/update API schemas + router pass num_ctx through.
- Frontend: optional numeric field on the Ollama credential form, with i18n
labels translated across all 13 locales.
- Docs: document the new default and the override under AI providers.
Closes#825
Header-based splitters (notably HTMLHeaderTextSplitter on complex pages
like Wikipedia or Project Gutenberg) can emit single-character or
punctuation-only chunks. Some embedding providers — including
llama.cpp's OpenAI-compatible endpoint — return null vector elements
for such inputs, which then crash response parsing in Esperanto with
'TypeError: float() argument must be a string or a real number, not
NoneType'.
chunk_text() now filters chunks below OPEN_NOTEBOOK_MIN_CHUNK_SIZE
tokens (default 5) after splitting. The filter is bypassed when it
would empty the result list, so legitimately short documents are
preserved.
* feat: make chunk sizing token-based with 512-token default
* fix: defer embedding debug token metrics
* chore: lower default chunk size to 400 tokens and document rationale
The previous 512-token default matched exactly the context window of
BERT-family embedders like mxbai-embed-large, leaving no margin for:
- tokenizer mismatch between our o200k_base measurement and the
embedder's own WordPiece tokenizer
- occasional splitter overshoot (RecursiveCharacterTextSplitter can
emit chunks slightly above chunk_size when separators are sparse)
- special tokens ([CLS], [SEP]) that consume context-window budget
400 tokens keeps ~20% headroom below 512 while still being a large
improvement over the old character-based default for most content.
Users with larger-context embedders can raise OPEN_NOTEBOOK_CHUNK_SIZE
via env var. Also adds a CHANGELOG entry for the full PR behavior
change.
* chore: move chunking changelog entry under 1.8.5
Target release is 1.8.5 — moving the Changed section out of Unreleased.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
- Catch only ValueError (decryption errors) instead of broad Exception
so NotFoundError and other failures propagate correctly
- Support migrate_to parameter in the fallback delete path so linked
models can be reassigned instead of always cascade-deleted
- Sanitize decryption_error message to not expose raw exception details
- Credential.get_all() now uses per-row error handling instead of failing on first bad row
- Broken credentials include decryption_error field with descriptive message
- DELETE endpoint falls back to direct DB delete when credential can't be decrypted
- Frontend shows amber warning alert for broken credentials with disabled test/edit/discover
- Added i18n translation keys for decryption error warning in all 9 locales
* fix: map base_url to endpoint for Azure credentials
The Azure credential form only exposes a base_url field, but the
connection tester, key provisioner, and Esperanto config all expect
an endpoint field. This maps base_url to endpoint for Azure providers
so credentials work without requiring a dedicated endpoint form field.
Closes#727
* docs: update Azure credential docs to reflect base_url mapping
- Bump esperanto dependency to >=2.20.0 for new provider profiles
- Register both providers in credentials, key provider, connection tester, model discovery, and models router
- Add frontend provider entries (display names, modalities, docs links)
- Add documentation sections for both providers in ai-providers.md, environment-reference.md, and provider comparison
- #627: Set source.asset (URL/file_path) before save() in async creation
path so failed sources are identifiable and retry works
- #670: Only overwrite source title if it's a placeholder ("Processing...")
or empty, preserving user-set custom titles
- #651: Cascade-delete linked models when credential is deleted instead of
returning 409 Conflict; remove unused delete_models parameter
- Add tests for all three fixes (12 new tests)
- Add .harness and .mcp.json to .gitignore
Broad 'except Exception' could silently swallow unexpected failures.
URLError and ConnectionError are both subclasses of OSError, so
'except (ImportError, OSError)' captures all real offline/not-installed
cases while letting genuine programming errors propagate.
Also include the exception detail in the warning message so failures
are diagnosable in logs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In air-gapped / offline Docker deployments, tiktoken.get_encoding() tries
to download the encoding file from openaipublic.blob.core.windows.net.
When that request fails it raises a URLError / OSError — not an ImportError
— so the previous except clause silently missed it and the crash surfaced in
the UI.
Widened `except ImportError` to `except Exception` so all failures —
"not installed" and "network unreachable" — fall through to the word-count
fallback (words × 1.3). Added a loguru WARNING so operators can see when
the fallback is active.
TIKTOKEN_CACHE_DIR now reads from the environment with a blank-safe
fallback (`or` guard prevents os.makedirs("") on empty env var). This lets
Docker images redirect the cache to a path outside /app/data/ so user-data
volume mounts cannot shadow the pre-baked encoding.
Both images now pre-download the o200k_base encoding during the builder
stage (internet is available at build time) and copy it into the runtime
image at /app/tiktoken-cache. ENV TIKTOKEN_CACHE_DIR=/app/tiktoken-cache
is set in the runtime stage so no network call is ever needed at runtime.
Added test_token_count_network_error_fallback in tests/test_utils.py:
patches tiktoken.get_encoding with a URLError and asserts token_count()
returns a positive int instead of raising.
Fixes#264
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(podcasts): integrate model registry for profiles and credential passthrough
Replace loose provider/model string fields with record<model> references
in podcast profiles, enabling credential passthrough to podcast-creator.
Backend:
- EpisodeProfile: outline_llm, transcript_llm (record<model>) replace
outline_provider/outline_model strings. New language field (BCP 47).
- SpeakerProfile: voice_model (record<model>) replaces tts_provider/
tts_model strings. Per-speaker voice_model override support.
- Migration 14: schema changes making legacy fields optional, adding new
record<model> fields.
- Data migration (migration.py): auto-converts legacy profiles to model
registry references on startup. Idempotent.
- podcast_commands.py: resolves credentials for ALL profiles before
calling podcast-creator.
- New /api/languages endpoint (pycountry + babel) with BCP 47 locale
codes (pt-BR, en-US, etc.).
Frontend:
- Episode/speaker profile forms use ModelSelector instead of manual
provider/model dropdowns.
- Language dropdown with BCP 47 codes in episode profile form.
- Per-speaker TTS voice model override in speaker profile form.
- "Templates" tab renamed to "Profiles".
- Setup required badge on unconfigured profiles.
- i18n updated across all 8 locales.
Closes#486, closes#552
* fix(i18n): remove unused legacy podcast provider/model keys
Remove 10 orphaned i18n keys across all 8 locales that were left behind
after replacing manual provider/model dropdowns with ModelSelector.
* fix: address review violations in podcast model registry
- P1: Remove profiles with failed model resolution from dicts to prevent
podcast-creator validation errors on unrelated profiles
- P2: Use centralized QUERY_KEYS.languages instead of inline key
- P3: Fix ISO 639-1 → BCP 47 in model field description and CLAUDE.md
- P3: Update "templates" → "profiles" in locale string values (all 8)
* chore: bump version to 1.8.0
- Add batching to generate_embeddings() (50 texts per batch with per-batch retry)
to prevent 413 Payload Too Large errors on large documents
- Add 413 error classification rule for user-friendly error messages
- Fix misleading "Created 0 embedded chunks" log in process_source_command
by removing premature get_embedded_chunks() call (embedding is fire-and-forget)
Closes#594
* docs: update CLAUDE.md and user docs for error handling and podcast retry
Add missing documentation for features introduced in v1.7.2 (#590) and
v1.7.3 (#595): error classification system, global exception handlers,
ConfigurationError, podcast failure recovery, and retry endpoint.
* chore: update uv.lock
* fix: surface podcast errors and enable retry for failed episodes
Fixes#335, #300
Re-raise exceptions in podcast command so surreal-commands marks jobs as
failed instead of completed. Surface error_message in API responses and
add a retry endpoint that deletes the failed episode and re-submits the
generation job. Frontend shows error details on failed episodes with a
retry button. Translations added for all 8 locales.
* fix: bump podcast-creator to >= 0.10
Fixes#302
* chore: release 1.7.3 - podcast failure recovery and retry
Bump podcast-creator to >= 0.11.2, disable automatic retries for
podcast generation to prevent duplicate episodes, and bump version
to 1.7.3.
Fixes#211, #218, #185, #355, #300, #302
* fix: resolve TypeScript error in handleRetry return type
Resolve conflicts in ask.py and chat.py by merging the try/except error
handling from main with the extract_text_content helper from the PR.
Also apply the same fix to source_chat.py and transformation.py which
had the same vulnerable isinstance/str() pattern for structured LLM
response content (e.g. Gemini's envelope format).
- Catch ConfigurationError alongside ValueError in get_default_model()
to preserve graceful fallback after ValueError→ConfigurationError migration
- Add _truncate() helper to error_classifier to cap pass-through and
default error messages at 200 chars, avoiding verbose internal details
Replace generic "An unexpected error occurred" messages with descriptive,
user-friendly error messages when LLM operations fail. Errors like invalid
API keys, wrong model names, and rate limits now surface clearly in the UI.
Adds error classification utility, global FastAPI exception handlers, and
frontend getApiErrorMessage() helper. Bumps version to 1.7.2.
Add empty-content validation in content_process() after extract_content()
returns. Sources with no extractable text (e.g. YouTube videos without
transcripts) now raise ValueError immediately instead of silently saving
an empty source. ValueError is already configured as a permanent failure
in the retry config, so no retries are wasted on unrecoverable situations.
Closes#527
Source.vectorize() wrapped its own ValueError in DatabaseOperationError,
bypassing the stop_on=[ValueError] retry guard in process_source_command.
This caused up to 15 retries when processing files with no extractable
text, blocking sync API requests indefinitely.
- Re-raise ValueError directly in Source.vectorize() instead of wrapping
- Add .strip() check to catch whitespace-only content
- Skip vectorization gracefully in save_source() when content is empty
- Add unit tests for vectorize error handling
Fixes#560
