* fix(sources): fall back to auto when a selected engine's runtime is absent
The content-processing engine choice is persisted in the database; the
runtime that serves it (Docling, local Crawl4AI) is installed on demand
from environment flags evaluated at boot. The two therefore drift: a
redeploy that drops OPEN_NOTEBOOK_ENABLE_CRAWL4AI/_DOCLING, a volume
moved to a new deployment, or a failed on-demand install all leave a
stored selection pointing at a runtime that is not there.
The source graph passed that selection straight to content-core, so
every affected extraction failed with "Could not extract any text
content from this source" - no mention of the engine, the runtime, or
the flag that would fix it. For a URL engine set to crawl4ai this breaks
URL ingestion entirely.
The graph now checks runtime availability before honoring the stored
engine and degrades to content-core's "auto" chain, logging a WARNING
that names the engine and the env var that would enable it. Engines with
no opt-in runtime (auto/simple/firecrawl/jina) are passed through
untouched.
The availability probes moved from api/routers/capabilities.py to
open_notebook/utils/runtime_capabilities.py so the graph can use them
without importing from the API layer; the capabilities endpoint keeps
identical behavior and its tests follow the probes to their new home.
Found by the smoke-e2e agent during v1.14.0 release testing, on a dev
environment that was in exactly this state. Pre-existing since v1.13.0
(#1122 made the runtimes opt-in, #432 made the stored selection take
effect), not a v1.14.0 regression.
* docs(changelog): record the unavailable-engine fallback fix
Surface content-core 2.x's docling_formulas (formula extraction) and
docling_vision (image/chart vision) enrichment flags in Settings →
Content Processing, mirroring the existing OCR toggle. Both default off
and are gated on Docling availability in the UI. The settings persist
via GET/PUT /api/settings and are threaded into content-core extraction
alongside docling_ocr. Migration 23 backfills the new fields on the
existing content_settings record. Labels and help are translated across
all 14 locales.
Closes#1131
Each transformation persists its own model_id, but neither call site that
invokes the transformation graph passed a config, so run_transformation()
always read a None model and fell back to the global default.
Forward transformation.model_id through the LangGraph `configurable` config
at both call sites (source-processing graph and the run_transformation
background command). An unset model_id remains None and falls back to the
default via the existing branch in provision_langchain_model().
Closes#1137
* feat(sources): log the effective content-core extraction engine
The source-processing graph now emits an INFO line naming the resolved
url_engine / document_engine / docling_ocr right before extraction.
content-core only logs its own engine dispatch at DEBUG, so operators
had no way to confirm which engine actually ran for a given source
(e.g. whether a persisted Crawl4AI selection took effect or the request
silently fell back to the 'auto' chain). Falls back to 'auto' in the log
when no override is set, matching content-core's default.
* docs(changelog): note the extraction-engine logging (#1125)
Expose content-core's docling_ocr flag as a user setting. OCR runs on
scanned PDFs and images when the Docling engine handles them; it's on by
default (matching content-core) and can be turned off for faster
processing of text-native documents.
- New `docling_ocr` boolean on ContentSettings (default True), plumbed
through the settings API and into ContentCoreConfig in the source graph.
- "Enable OCR" checkbox in the Content Processing settings card, with
label + help translated across all 14 locales.
Part of #939. Stacked on #432.
* feat(sources): add Crawl4AI URL engine and honor persisted engine settings (#432)
- Add "crawl4ai" as a selectable URL processing engine (domain Literal,
settings API validation, SettingsForm select, and label across all 14
locales; urlHelp updated in en-US to describe the new fallback chain).
- The source graph now loads the persisted ContentSettings and passes the
document/URL engine choices to ContentCoreConfig. Previously it built a
hard-coded ContentSettings with "auto" engines, so a user's selection in
Settings never took effect. Falls back to defaults if settings can't load.
- Crawl4AI Docker mode is driven by content-core's native CRAWL4AI_API_URL
env var (documented separately under #1105).
Part of #939.
* fix(432): bundle Crawl4AI runtime + address review
- Bundle the Crawl4AI runtime so its local, no-API-key mode works out of
the box: depend on content-core[crawl4ai] and install the Chromium
browser via playwright in the Docker runtime-base (both image variants).
Footprint is modest (no torch/transformers/CUDA); image grows ~300 MB
from Chromium + system libs.
- Preserve the server-side traceback when persisted content settings fail
to load (logger.opt(exception=True)) instead of only the message.
- Reset the ContentSettings singleton between domain tests (clear_instance)
so a non-default value can't leak into neighboring tests.
Addresses review on #432.
* i18n(432): translate urlHelp Crawl4AI description across all 13 non-en locales
The Crawl4AI engine label was already localized; this brings the URL-engine
'help me choose' text in line with en-US in every locale — describing
Crawl4AI (local JS rendering, no API key) and its place in the auto
fallback chain (Firecrawl -> Jina -> Crawl4AI -> simple).
* chore(sources): upgrade to content-core 2.x
Bump content-core 1.14.x -> 2.0.4 and adapt the source graph to the new
keyword-only extract_content API.
- extract_content is now keyword-only; engine/model overrides move to
ContentCoreConfig (audio model still sourced from Default Models).
- ProcessSourceState was removed; the graph now consumes ExtractionOutput,
which no longer echoes url/file_path back — carry those from the input
state into the saved Asset.
- content-core 2.x no longer deletes the uploaded source file, so honor
delete_source on our side after a successful extraction.
- drop the obsolete output_format param (markdown is the default).
pymupdf (AGPL) is replaced transitively by pdfplumber (MIT); moviepy is
gone (direct ffmpeg), which fixes MP3-with-chapters audio.
Part of #939.
* docs(changelog): note content-core 2.x upgrade (#1103)
* fix(sources): wire YouTube transcript language preferences into ContentCoreConfig
content-core's default youtube_languages is only en/es/pt. Pass the broader
list Open Notebook has always intended so non-English videos still resolve a
transcript. (cubic P2 on #1116)
* ci: gate PRs on mypy, start ignore_errors burn-down
Add a backend-typecheck CI job running uv run python -m mypy . and bring
the repo-wide baseline from 197 errors to 0 so the gate blocks new type
errors from now on:
- enable the pydantic mypy plugin (resolves 138 false positives on
models whose fields have Field(None, ...) defaults)
- fix the remaining errors with real annotations; the only new type:
ignore comments cover a genuine langgraph typing limitation (partial
state dicts are valid at runtime but the overloads require the full
state type) and tests that intentionally pass invalid input
- start the ignore_errors burn-down: open_notebook.graphs.transformation,
open_notebook.graphs.ask and api.routers.models are now type-checked;
stale blocks for the deleted api.client and api.podcast_api_service
modules removed. Only open_notebook.domain.notebook remains exempt
(DB layer is migrating to surreal-basics)
- fix the mypy.ini header comment to describe what the config does
* test: use typing.get_args on the registry literal check
The registry test landed in parallel using Literal.__args__, which the
mypy gate in this branch rejects; align it with the get_args() idiom
used by the rest of the file.
* chore(release): prepare v1.10.0
* style: fix import ordering flagged by ruff
* fix(sources): return 404 for missing source and fix retry 500 from double-prefixed command id
- GET /sources/{id} mapped NotFoundError to a generic 500; now returns 404
- POST /sources/{id}/retry double-prefixed the command id (command:command:...),
raising 'too many values to unpack' after queuing; align with the create path
- update retry test mock to realistic prefixed command id + guard against
double-prefix; add 404 regression test
* fix(sources): mark failed extraction as failed so retry surfaces (#726)
content-core signals soft failures (unreachable/invalid URL) by returning
title=Error + 'Failed to extract content:' body instead of raising, and the
process_source command swallowed permanent ValueErrors into a success=False
result. Since surreal-commands marks a job completed when the function returns,
failed ingests showed status 'completed' and never offered the retry button.
- source graph: detect the content-core failure sentinel and raise
- process_source_command: re-raise ValueError (stop_on already prevents retry)
so the job is marked failed and the source becomes retryable
* feat(notebook): per-type bulk context actions for sources and notes (#223)
- Sources context menu now offers 'insights only' (sources without insights
are excluded rather than forced to full), 'full content', and 'exclude all'
- Add the same bulk Context menu to the Notes column (include all / exclude all)
- Bulk choices propagate to items loaded later via pagination
- New locale keys (includeAllInsights/includeAllFull) across all 14 locales
- Unit tests for the new bulk modes and note context helpers
- #627: Set source.asset (URL/file_path) before save() in async creation
path so failed sources are identifiable and retry works
- #670: Only overwrite source title if it's a placeholder ("Processing...")
or empty, preserving user-set custom titles
- #651: Cascade-delete linked models when credential is deleted instead of
returning 409 Conflict; remove unused delete_models parameter
- Add tests for all three fixes (12 new tests)
- Add .harness and .mcp.json to .gitignore
Add empty-content validation in content_process() after extract_content()
returns. Sources with no extractable text (e.g. YouTube videos without
transcripts) now raise ValueError immediately instead of silently saving
an empty source. ValueError is already configured as a permanent failure
in the retry config, so no retries are wasted on unrecoverable situations.
Closes#527
Source.vectorize() wrapped its own ValueError in DatabaseOperationError,
bypassing the stop_on=[ValueError] retry guard in process_source_command.
This caused up to 15 retries when processing files with no extractable
text, blocking sync API requests indefinitely.
- Re-raise ValueError directly in Source.vectorize() instead of wrapping
- Add .strip() check to catch whitespace-only content
- Skip vectorization gracefully in save_source() when content is empty
- Add unit tests for vectorize error handling
Fixes#560
* chore: improve podcast transcripts
* fix: remove date from insight - fixes#241
* fix: improve scrolling on source and insights - fixes#237
* chore: update esperanto to fix: #234
* chore: update esperanto to fix#226
* fix: process vectorization as subcommands to handle larger documents more gracefully - fix: #229
* feat: enable background job retry capabilities
* feat: reenable content types that were disabled during alpha version
* fix: remove unnecessary model caching causing many issues.
* feat: support multiple azure endpoints and keys just like openai compatible. Fixes#215
* docs: update azure variables
* chore: bump and update dependencies
New front-end
Launch Chat API
Manage Sources
Enable re-embedding of all contents
Sources can be added without a notebook now
Improved settings
Enable model selector on all chats
Background processing for better experience
Dark mode
Improved Notes
Improved Docs:
- Remove all Streamlit references from documentation
- Update deployment guides with React frontend setup
- Fix Docker environment variables format (SURREAL_URL, SURREAL_PASSWORD)
- Update docker image tag from :latest to :v1-latest
- Change navigation references (Settings → Models to just Models)
- Update development setup to include frontend npm commands
- Add MIGRATION.md guide for users upgrading from Streamlit
- Update quick-start guide with correct environment variables
- Add port 5055 documentation for API access
- Update project structure to reflect frontend/ directory
- Remove outdated source-chat documentation files
Creates the API layer for Open Notebook
Creates a services API gateway for the Streamlit front-end
Migrates the SurrealDB SDK to the official one
Change all database calls to async
New podcast framework supporting multiple speaker configurations
Implement the surreal-commands library for async processing
Improve docker image and docker-compose configurations
