* feat(frontend): design foundation — Quiet Green tokens, fonts, primitives, /dev/design
- Port the ratified design token system into globals.css: core palette,
semantic surfaces/ink/hairlines, content-type hues, evidence classes,
context states, squared 4-6px radii, shadow scale (popovers own the
only real shadow)
- Swap Inter for Bricolage Grotesque (display) + Instrument Sans (UI) +
Spline Sans Mono (data) via next/font
- Restyle shadcn primitives to the foundation laws: fern primary button,
red reserved for destructive, hairline cards without default shadow,
underline tabs, quiet neutral badges, teal focus ring, single-shadow
overlays
- Add /dev/design living styleguide (dev-only, 404 in production)
rendering all tokens and primitives with a theme toggle
* refactor(frontend): sweep hard-coded palette colors into design tokens
Replace raw Tailwind palette classes (red/amber/green/blue/emerald/sky/
purple…) across 38 components with the semantic token utilities:
errors -> destructive, warnings -> warn (clay), success -> fern,
system/info -> teal, context states -> ctx-* tokens, model-type chips
redistributed over the owned palette. Light/dark duplicates collapse
into single token classes since tokens re-resolve per theme.
* docs: changelog entry for the Quiet Green design foundation
* feat: add 'add source' button to sources page
* feat(i18n): add sources.newSource translation to all locales
* refactor(frontend): deduplicate AddSourceDialog in sources page
* i18n(de-DE): add missing sources.newSource key
The new sources.newSource key was added to the other 13 locales but not
de-DE, so the locale parity test (src/lib/locales/index.test.ts) failed
and de-DE users would fall back to the raw key. Add the German string.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: prevent Ask streaming loading from getting stuck on dangling connections
Adds a safety-net timeout (5 min) to useAsk that forces the loading
state to end even when the SSE stream's 'done' signal doesn't reach
the browser — which happens in Docker/standalone deployments where
the Next.js proxy connection is left open without proper termination.
Changes:
- use-ask.ts: AbortController per request, mountedRef for cleanup,
STREAM_TIMEOUT_MS that resets isStreaming, stopStreaming() helper
that centralizes cleanup + clears the timeout
- search.ts: pass AbortSignal through to fetch() so the timeout can
actually cancel the HTTP request
The complete event now also carries final_answer content as a
backup path in case the final_answer event is missed.
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: reduce stream timeout from 5 min to 75s
* fix(ask): make stream watchdog idle-based and abort dangling connections
The Ask stream safety timeout was a fixed 75s wall-clock cap measured from
stream start and never reset by incoming SSE events. A legitimately slow but
healthy Ask (slow local models — Ollama, LM Studio) would have its loading
state killed mid-stream, showing a partial/empty answer while the backend was
still producing — the exact slow-hardware scenario in #777.
- Re-arm the timeout on every received chunk so it fires only when the
connection goes genuinely silent (idle timeout, not wall-clock).
- Align the idle window to the API client budget (NEXT_PUBLIC_API_TIMEOUT_MS,
default 10 min; 0 disables) so slow local models aren't cut off. Export
API_TIMEOUT_MS from the client for this.
- On fire, abort the in-flight request so a truly dangling connection frees the
socket/reader instead of leaking it (the catch already swallows AbortError).
---------
Co-authored-by: dyzur <dyzur@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix: prevent source dialog action overlap
* fix(sources): pad only the right of the detail header to clear close button
px-12 cleared the overlap but also indented the header's left edge 40px
past the tab content below (which stays px-2), misaligning the title. Keep
px-2 and add pr-12 so the header left-aligns with the body while the actions
menu still clears the dialog close button on the right.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
* fix(frontend): support allowedDevOrigins via env var
Accessing the Next.js dev server from a LAN IP fails all HMR/dev
resource requests: Next.js blocks cross-origin dev requests (including
the webpack-hmr websocket) from any host not in an explicit allowlist,
as a DNS-rebinding guard. Server log:
Blocked cross-origin request to Next.js dev resource
/_next/webpack-hmr from "<lan-ip>".
Add NEXT_ALLOWED_DEV_ORIGINS (comma-separated hostnames) to populate
Next's allowedDevOrigins config, so LAN/custom-hostname dev access can
be enabled per-deployment without hardcoding a host into next.config.ts.
* fix(frontend): treat whitespace-only NEXT_ALLOWED_DEV_ORIGINS as unset
A whitespace-only or comma-only value parsed to an empty array, which is
truthy and still spread allowedDevOrigins: [] into the config, overriding
Next's default dev-origin behavior. Only set the key when at least one
origin survives trimming.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
.env.example, development-setup.md, and examples/README.md referenced
APP_PASSWORD and BASIC_AUTH_USERNAME/BASIC_AUTH_PASSWORD, none of which
exist in code. api/auth.py's PasswordAuthMiddleware only reads
OPEN_NOTEBOOK_PASSWORD; the old names silently did nothing.
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Retro improvements from the v1.14.0 release:
- release-image-test.sh gains a `probe` scenario (part of `all`) that
runs the two container-level checks done by hand during v1.14.0:
OPEN_NOTEBOOK_WORKER_MAX_TASKS reaching the in-image worker (#1141's
supervisord `sh -c` expansion), and the worker surviving startup with
HTTP_PROXY set while the user's NO_PROXY value is preserved (#1160,
internal SurrealDB websocket not tunneled). Self-contained standalone
containers with their own cleanup, so a probe failure can't leak into
the fresh/upgrade scenarios.
- RELEASE_PROCESS.md documents two gotchas the release hit: never leave
the version bump uncommitted on a branch (it leaked into an unrelated
fix PR), and a post-tag fix requires a full re-cut (move tag + rebuild
images), not just a tag nudge, or publication promotes the un-fixed
artifact to v1-latest.
Verified: `release-image-test.sh probe lfnovo/open_notebook:1.14.0`
passes 4/4.
The episode profile "Max output tokens" field (#991) confused users:
nothing explained what it controls, and whether it was a
per-conversation, per-turn, or per-call limit. An empty field also
showed "Must be a positive integer" — a scary error on an optional
field — because a transient/empty number-input state produced NaN,
which slipped past the nullable() guard into the positive() rule.
- Add help text (all 14 locales) stating it caps output tokens per
generation step — the outline and the full transcript, not a per-turn
or conversation limit — and that blank uses the built-in defaults
(3000 outline / 5000 transcript, from podcast-creator).
- Add a placeholder ("Leave blank for defaults") so the empty field
reads as intentionally optional rather than unfilled.
- Map empty and NaN inputs to null in setValueAs, so an empty or
half-typed value reads as "unset" and never trips the positive-integer
rule. Genuinely invalid entries (0, negatives, decimals) still error.
Field stays an optional override — no behavior change to generation.
Verified in the browser: help renders, placeholder shows, empty field
has no error, zero console errors. Frontend lint/test/build green.
* fix(sources): fall back to auto when a selected engine's runtime is absent
The content-processing engine choice is persisted in the database; the
runtime that serves it (Docling, local Crawl4AI) is installed on demand
from environment flags evaluated at boot. The two therefore drift: a
redeploy that drops OPEN_NOTEBOOK_ENABLE_CRAWL4AI/_DOCLING, a volume
moved to a new deployment, or a failed on-demand install all leave a
stored selection pointing at a runtime that is not there.
The source graph passed that selection straight to content-core, so
every affected extraction failed with "Could not extract any text
content from this source" - no mention of the engine, the runtime, or
the flag that would fix it. For a URL engine set to crawl4ai this breaks
URL ingestion entirely.
The graph now checks runtime availability before honoring the stored
engine and degrades to content-core's "auto" chain, logging a WARNING
that names the engine and the env var that would enable it. Engines with
no opt-in runtime (auto/simple/firecrawl/jina) are passed through
untouched.
The availability probes moved from api/routers/capabilities.py to
open_notebook/utils/runtime_capabilities.py so the graph can use them
without importing from the API layer; the capabilities endpoint keeps
identical behavior and its tests follow the probes to their new home.
Found by the smoke-e2e agent during v1.14.0 release testing, on a dev
environment that was in exactly this state. Pre-existing since v1.13.0
(#1122 made the runtimes opt-in, #432 made the stored selection take
effect), not a v1.14.0 regression.
* docs(changelog): record the unavailable-engine fallback fix
The DNS-pinning guard added in #1063 is the highest-risk change in the
v1.14.0 release for self-hosters: it sits on the outbound path of every
credential save, connection test and model discovery, so a regression
that over-blocks silently costs users their provider.
TestPinnedHttpTarget already covers what the guard must reject. This
adds the inverse assertions - the deployment shapes that must keep
working - which prepare_pinned_http_target had no coverage for at all
(the existing legitimate-use tests only exercised validate_url, which
is not what gates outbound requests):
- Ollama on localhost, including IPv4 preference when both families
resolve
- host.docker.internal (containerized app -> host service)
- private LAN by IP literal (LM Studio) and by hostname
- IPv6 loopback literal
- Tailscale CGNAT space (100.64.0.0/10) - shared, not link-local
- AAAA-only endpoints producing a bracketed, parseable URL
- query strings surviving the rewrite (PPQ's ?type=all discovery URL)
All pass against the current implementation - this pins the behavior
rather than fixing a defect.
The release audit (git log v1.13.0..main) found 27 merged commits but
only 11 represented in [Unreleased]. Adds the 14 missing entries and
corrects two existing problems:
- The #1137 entry (per-transformation model selection) was filed under
[1.13.0], but its commit is not in the v1.13.0 tag — v1.13.0 shipped a
changelog entry for an unshipped fix. Moved verbatim to [Unreleased].
- The anthropic_compatible entry was a one-line stub with no issue
reference; rewritten in house style and pointed at #675.
New entries cover the DNS-pinning hardening (#1063), worker concurrency
(#893), episode max_tokens in the UI (#991), error-text capping (#1136),
the IME/chat composer perf fix (#1147), the anthropic shim removal
(#1055), the esperanto floor bump (#1176), ja-JP translation quality
(#998), i18n placeholder parity (#1159), the OLLAMA_API_BASE docs fix
(#1148), and the CI/tooling chores (#1134, #940, #1133, #1158).
A ### Changed section was added; every commit in the range is now
represented.
Bump two dependencies flagged by Dependabot ahead of the v1.14.0 release:
- mcp 1.25.0 -> 1.28.1 (3 high alerts). Transitive via content-core ->
fastmcp; the advisories cover MCP *server* transports (missing
Host/Origin validation on the WebSocket transport, HTTP transports
serving session requests without verifying the authenticated
principal, and experimental task handlers exposing other clients'
tasks). Open Notebook does not serve MCP, so none of them are
reachable here, but fastmcp accepts the newer release and the bump
resolves cleanly with no other dependency churn.
- axios 1.16.0 -> 1.18.1 (medium alert). Prototype pollution allowing
Basic auth injection through auth subfields. Direct frontend
dependency; pulls nested agent-base/https-proxy-agent, which newer
axios uses for proxy handling.
Verified: uv sync + content-core/fastmcp import clean, frontend
npm ci + npm run build green, npm audit reports 0 vulnerabilities.
Surface content-core 2.x's docling_formulas (formula extraction) and
docling_vision (image/chart vision) enrichment flags in Settings →
Content Processing, mirroring the existing OCR toggle. Both default off
and are gated on Docling availability in the UI. The settings persist
via GET/PUT /api/settings and are threaded into content-core extraction
alongside docling_ocr. Migration 23 backfills the new fields on the
existing content_settings record. Labels and help are translated across
all 14 locales.
Closes#1131
* fix(proxy): keep internal SurrealDB websocket out of HTTP proxy (#1160)
websockets 15.0 auto-detects HTTP_PROXY/HTTPS_PROXY and tunnels even
ws:// connections through the proxy. The SurrealDB SDK connects over a
websocket, so with a proxy set the internal DB connection was routed
through the external proxy, which rejected the internal host with HTTP
403 and killed the worker/API on startup.
- Add ensure_internal_no_proxy() helper that merges host.docker.internal,
surrealdb, localhost, 127.0.0.1 into no_proxy/NO_PROXY (never clobbering
a user value) and call it at API, worker and DB-module startup.
- Add host.docker.internal and surrealdb to the .env.example and docs
NO_PROXY examples.
- Add unit tests for the injection helper.
* fix(proxy): preserve NO_PROXY wildcard and include custom SurrealDB host (#1160)
Address review findings on the no_proxy injection:
- NO_PROXY=* (bypass all hosts) is now treated as terminal: leave the
user's config untouched instead of narrowing the wildcard to a finite
list by appending the internal hosts.
- Parse the SurrealDB host from SURREAL_URL (falling back to
SURREAL_ADDRESS) and add it to the bypass list, so deployments with a
custom DB host/IP no longer route DB traffic through the proxy. Unset
or malformed values fall back to the four defaults gracefully.
- Drop the inaccurate getproxies() caching remark in the test.
* fix(models): stop auto-assign from re-filling cleared optional defaults
Auto-assign treated every empty default slot as "missing" and filled it,
so an optional slot a user deliberately cleared (to fall back to the chat
model) got silently re-populated on the next run, undoing the intent.
- Auto-assign now fills only the required slots (chat, embedding); the
optional slots (transformation, tools, large context, TTS, STT) are
left untouched.
- get_default_model("large_context") now falls back to the chat model
when unset, matching transformation/tools (TTS/STT still return None).
- Settings UI shows an inline hint on each empty optional slot: the text
slots show "using chat model (<name>)"; TTS/STT show a not-configured
hint. Required slots remain non-clearable. New i18n keys across all 14
locales.
Closes#1098
* test: guard await_args against None for mypy
The locale parity test and TranslationShape type only compared keys, not
the interpolation placeholders inside values. This let single-brace drift
(a stray {count} instead of i18next's {{count}}) slip through unnoticed.
Extend the parity suite to, for every leaf string, compare the set of
{{...}} placeholders against en-US and flag any stray single-brace token
that maps to a real en-US placeholder. Aligns zh-CN/zh-TW
podcasts.generationStartedDesc, which referenced a {{name}} placeholder
absent from the en-US reference.
PPQ's bare /v1/models returns only chat/language models; ?type=all is
required to also list the embedding, STT and TTS models it advertises as
a multi-modality gateway. Without it those modalities never surfaced in
discovery. PPQ_MODEL_TYPES already classifies the returned ids.
Closes#1180
esperanto 2.25.1 forwards a custom base_url for anthropic_compatible
models natively in to_langchain(), so the ChatAnthropic re-injection
shim added by #1043 is no longer needed.
- provision.py: drop _to_langchain() helper; call model.to_langchain()
- models.py: drop the _open_notebook_provider marker (shim-only consumer)
- tests: assert base_url forwarding via the native to_langchain() path
Closes#1055
Register anthropic_compatible in the provider registry so its env config,
modalities, test model, and /api/providers entry are derived from PROVIDERS
(#1075's single source of truth); the only manual copy is the SupportedProvider
Literal. Maps to esperanto's anthropic provider with a custom base_url, and
re-injects that base_url via ChatAnthropic since esperanto's to_langchain drops
it. Connection-test and model discovery mirror the openai_compatible siblings,
including DNS-rebinding pinning (prepare_pinned_http_target). A single shared
validator enforces the base_url + api_key requirement on both the create and
update paths.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Expose Esperanto's built-in omlx OpenAICompatibleProfile in Settings with
Ollama-style UX (default http://localhost:11435/v1, optional API key,
language+embedding discovery). No openai-compatible remapping or
OPENAI_COMPATIBLE_* env mirroring.
Fixes#1048
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Wire four esperanto 2.25.1 providers into the provider matrix:
- Cohere (COHERE_API_KEY): language + embedding via the native v2 API,
with a bespoke discoverer (AIFactory.get_provider_models) since Cohere
is not OpenAI-compatible. Reranking is out of scope (#1087).
- Deepgram: extend modalities to add speech_to_text (Nova/Whisper)
alongside the existing Aura text_to_speech voices.
- PayPerQ / PPQ (PPQ_API_KEY): multi-modality OpenAI-compatible gateway
(https://api.ppq.ai/v1), auto-discovered via the /models endpoint.
- Novita (NOVITA_API_KEY): OpenAI-compatible LLM gateway
(https://api.novita.ai/openai), auto-discovered via /models.
Registry-derived surfaces (env config, modalities, test models, discovery
table, GET /api/providers) update automatically; the SupportedProvider
Literal, key_provider config, availability env map, docs and tests were
updated to match. Closes#1170
esperanto 2.25.0 added OpenRouter for TTS and STT (previously LLM/embedding
only). Expose those modalities in Open Notebook.
- provider_registry: openrouter modalities -> all four (adds speech_to_text,
text_to_speech)
- model_discovery: bespoke discover_openrouter_models combines the live
OpenAI-compatible /models listing with a static seed of the audio model ids
esperanto ships as defaults (microsoft/mai-voice-2 for TTS, openai/whisper-1
and openai/whisper-large-v3 for STT), since OpenRouter's listing does not
reliably tag audio models. Seed only when live discovery returns models, so a
failed discovery never registers unusable audio models.
- credentials_service: credential-based discovery seeds the same audio ids for
openrouter (only after a successful /models response)
- connection_tester: exclude openrouter from DEFAULT_TEST_VOICES so TTS tests
use esperanto's model-specific available_voices (voices are model-specific;
the default model uses Microsoft neural voice names, not alloy/nova)
- tests, docs (provider matrix, AI provider guides) and CHANGELOG
Closes#987
Credential.to_esperanto_config() emitted the generic project/location
keys for all providers, but esperanto's Vertex providers accept
vertex_project/vertex_location. Credential-linked (non-env) Vertex TTS
therefore crashed with "__init__() got an unexpected keyword argument
'project'". Map to the Vertex-specific key names for the vertex provider
only; the Credential schema/API fields stay project/location and
non-Vertex providers are unaffected.
Closes#1151
* fix(notebook): cascade-delete chat sessions on notebook deletion
Deleting a notebook removed its notes and exclusive sources but left
chat_session records orphaned. Extend Notebook.delete() to enumerate the
notebook's chat sessions via the existing refers_to relation and delete
each one, and report chat_session_count in the delete preview.
Closes#1124
* refactor(notebook): drop chat-session count from delete preview
The delete-preview API advertised chat_session_count but the frontend
dialog and locales never render it, creating a UI inconsistency. Trim
the preview back to notes and sources; the deletion cascade and its
post-delete deleted_chat_sessions count are unchanged.
* perf(chat): isolate composer state to fix IME lag in long chats
Move the chat composer input state into a dedicated ChatComposer child so
keystrokes (including IME composition) no longer re-render the whole message
history. Memoize each message row with React.memo and make the reference-click
handler stable with useCallback so historical messages don't re-render on
composer input. Send behavior and the Ctrl/Cmd+Enter shortcut are preserved.
Closes#1147
* fix(chat): stabilize modal callbacks and use realistic shortcut tests
Memoize openModal/closeModal in useModalManager with useCallback so their
identities stay stable across renders. This keeps handleReferenceClick (which
depends on openModal) stable, so the memoized ChatMessage rows no longer
re-render on every ChatPanel render — completing the IME-lag fix.
Split the Ctrl/Cmd+Enter test into two realistic paths (Mac metaKey, non-Mac
ctrlKey) driven by a mocked userAgent, instead of an impossible combined
ctrlKey+metaKey event.
Each transformation persists its own model_id, but neither call site that
invokes the transformation graph passed a config, so run_transformation()
always read a None model and fell back to the global default.
Forward transformation.model_id through the LangGraph `configurable` config
at both call sites (source-processing graph and the run_transformation
background command). An unset model_id remains None and falls back to the
default via the existing branch in provision_langchain_model().
Closes#1137
The quick-start guide and the Ollama compose examples set OLLAMA_BASE_URL,
but no code path reads that variable. The authoritative env var is
OLLAMA_API_BASE (see open_notebook/ai/provider_registry.py). Following the
examples verbatim left Ollama permanently unavailable with no error.
Renames OLLAMA_BASE_URL to OLLAMA_API_BASE in the three remaining shipped
files, keeping the value unchanged.
* feat(models): offer newer ElevenLabs models in discovery
Add eleven_v3, eleven_flash_v2_5, eleven_flash_v2 (TTS) and scribe_v2 (STT)
to ElevenLabs model discovery. Esperanto passes model_id through to the
ElevenLabs API, so no provider changes are required.
* docs(changelog): reference PR #1167
* fix(i18n): improve Japanese translation quality
- Replace 'トランスフォーメーション' with natural '変換'
- Improve various phrasing for natural Japanese
- Add spacing in duration format
- Refine accessibility labels and error messages
* fix(i18n): remove redundant 含める in ja-JP includeAllInsights
---------
Co-authored-by: Lovin <lovin@example.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
build-regular is a required status check that ran a full (several-minute)
Docker build on every PR, including docs/config-only PRs where the image is
byte-for-byte identical (#1134, surfaced merging #1129).
Add a paths-filter 'changes' job and gate build-regular's build steps on
'push to main OR image-affecting paths changed'. The job itself always runs,
so the required check always reports — avoiding the 'path-filtered required
check never reports -> merge deadlock' trap (a workflow-level paths filter
would have caused exactly that). Docs/config PRs now report build-regular
green in seconds; PRs touching Dockerfile/deps/app/frontend build as before.
Release build (build-and-release) is untouched.
Closes#1134
The worker processes all queued tasks up to a fixed concurrency of 5, which
overloads single-GPU / local-LLM setups and triggers rate limits (#893). The
surreal-commands worker already accepts --max-tasks; wire it to a new
OPEN_NOTEBOOK_WORKER_MAX_TASKS env var (default 5, set 1 for sequential) at
every launch point.
- Makefile (worker-start, start-all): --max-tasks "$${VAR:-5}" (Make-escaped)
- dev-init.sh: POSIX ${VAR:-5} default expansion
- supervisord.conf: wrap in sh -c so the shell expands the var (command= does
not run through a shell)
- docker-compose.yml: commented environment example
- .env.example + environment-reference.md: document it, incl. the launch-time
sourcing behavior
Also fixes a phantom doc entry: SURREAL_COMMANDS_MAX_TASKS was documented but
is not read anywhere (the worker's --max-tasks Typer option has no envvar);
replaced with the real OPEN_NOTEBOOK_WORKER_MAX_TASKS.
Closes#893
* fix(sources): cap error text surfaced to clients
Source processing status (get_source_status) and sync-processing failures
returned the raw command/result error_message to clients unbounded, which
could leak arbitrary internal exception text and didn't match the error
capping applied elsewhere in the API.
Add a None-safe _truncate_error helper (200-char cap, ellipsis when cut)
and apply it on both paths. Adds focused unit tests for the helper.
Closes#1136
* test: narrow Optional return before assertions to satisfy mypy
Adds an optional "Max output tokens" number input to the episode profile form, mirroring num_segments handling: zod validation (positive integer, clearable to null), form defaults/reset, and submit payload. Shows the value in episode card details and adds max_tokens to the episode profile types.
Includes maxTokens/maxTokensInteger/maxTokensPositive translation keys in all 14 locales (the locale parity test requires new keys everywhere).
Closes#991
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix: pin DNS for outbound provider HTTP requests
Close the DNS-rebinding TOCTOU left by validate_url alone by resolving
once and connecting to the vetted IP (Host/SNI preserved). Apply
consistently to openai_compatible, ollama, and azure discovery/test paths.
* fix: restore @dataclass newline in model_discovery
* fix: resolve mypy failure and pin openai custom base_url discovery
- Cast getaddrinfo sockaddr[0] to str in _resolve_safe_ips so the
list[str] append typechecks (mypy CI gate was failing).
- Apply DNS pinning to the openai provider's user-supplied base_url
discovery path, mirroring the openai_compatible path, so httpx cannot
re-resolve to a metadata address after validation (DNS-rebinding TOCTOU).
- Add a test asserting the pinned IP URL, Host header and SNI extension
reach httpx on the openai base_url discovery path.
---------
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Use OLLAMA_API_BASE (the variable Open Notebook reads) instead of the
outdated OLLAMA_BASE_URL placeholder.
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Create .pre-commit-config.yaml with ruff (lint + format), mypy and basic
hygiene hooks so local commits fail for the same reasons PRs would.
- Ruff lint (--fix for local convenience, no drift vs CI's ruff check .)
- Ruff format check (not yet gated in CI but zero drift — same tool)
- Mypy via language:system/uv to match CI's uv run python -m mypy .
- Hygiene: large files, merge conflicts, YAML/TOML syntax, trailing
whitespace, EOF newlines
- Updated development-setup.md with install/run/skip/update docs
Closes#940
Co-authored-by: dyzur <dyzur@users.noreply.github.com>
Co-authored-by: Luis Novo <lfnovo@gmail.com>
Retro improvements from the v1.13.0 release:
- rc-stack.sh 'up' now docker-pulls lfnovo/open_notebook:<tag> by default so a
same-named local build can't shadow the registry artifact during Phase 6
verification (non-fatal for local-only tags).
- New --with-runtimes flag enables the opt-in heavy engines (Docling +
Crawl4AI) on the RC stack via RC_ENABLE_DOCLING/RC_ENABLE_CRAWL4AI, so the
published image's opt-in install path can be exercised with real data. The
release-test compose defaults these empty, keeping fresh/upgrade tests fast.
- Documented both gotchas (local-tag shadowing; judging opt-in gating on a
clean image vs a dev venv with runtimes installed out-of-band) in
RELEASE_PROCESS.md.
Bump version to 1.13.0 and date the changelog section. Also close the two
remaining changelog gaps for changes merged since v1.12.0:
- #1058: podcast generation honors speaker_profile in generate_podcast_command
- #999: transformation create works from the empty state
* feat(sources): log the effective content-core extraction engine
The source-processing graph now emits an INFO line naming the resolved
url_engine / document_engine / docling_ocr right before extraction.
content-core only logs its own engine dispatch at DEBUG, so operators
had no way to confirm which engine actually ran for a given source
(e.g. whether a persisted Crawl4AI selection took effect or the request
silently fell back to the 'auto' chain). Falls back to 'auto' in the log
when no override is set, matching content-core's default.
* docs(changelog): note the extraction-engine logging (#1125)
Docling and local Crawl4AI are now opt-in, installed on first container startup
via OPEN_NOTEBOOK_ENABLE_DOCLING / OPEN_NOTEBOOK_ENABLE_CRAWL4AI, keeping the
default image lean (Crawl4AI un-bundled). Downloads cache on the /app/data
volume; installs are blocking with loud logs and degrade-don't-die on failure.
A GET /api/capabilities probe reports actual availability and the Settings UI
gates the Docling/Crawl4AI engines and OCR toggle (with env-var hints) until the
runtime is present. Docs folded from #1121 and rewritten to opt-in; ADR-007 added.
Closes#1122. Closes#1105.
Expose content-core's docling_ocr flag as a user setting. OCR runs on
scanned PDFs and images when the Docling engine handles them; it's on by
default (matching content-core) and can be turned off for faster
processing of text-native documents.
- New `docling_ocr` boolean on ContentSettings (default True), plumbed
through the settings API and into ContentCoreConfig in the source graph.
- "Enable OCR" checkbox in the Content Processing settings card, with
label + help translated across all 14 locales.
Part of #939. Stacked on #432.
* feat(sources): add Crawl4AI URL engine and honor persisted engine settings (#432)
- Add "crawl4ai" as a selectable URL processing engine (domain Literal,
settings API validation, SettingsForm select, and label across all 14
locales; urlHelp updated in en-US to describe the new fallback chain).
- The source graph now loads the persisted ContentSettings and passes the
document/URL engine choices to ContentCoreConfig. Previously it built a
hard-coded ContentSettings with "auto" engines, so a user's selection in
Settings never took effect. Falls back to defaults if settings can't load.
- Crawl4AI Docker mode is driven by content-core's native CRAWL4AI_API_URL
env var (documented separately under #1105).
Part of #939.
* fix(432): bundle Crawl4AI runtime + address review
- Bundle the Crawl4AI runtime so its local, no-API-key mode works out of
the box: depend on content-core[crawl4ai] and install the Chromium
browser via playwright in the Docker runtime-base (both image variants).
Footprint is modest (no torch/transformers/CUDA); image grows ~300 MB
from Chromium + system libs.
- Preserve the server-side traceback when persisted content settings fail
to load (logger.opt(exception=True)) instead of only the message.
- Reset the ContentSettings singleton between domain tests (clear_instance)
so a non-default value can't leak into neighboring tests.
Addresses review on #432.
* i18n(432): translate urlHelp Crawl4AI description across all 13 non-en locales
The Crawl4AI engine label was already localized; this brings the URL-engine
'help me choose' text in line with en-US in every locale — describing
Crawl4AI (local JS rendering, no API key) and its place in the auto
fallback chain (Firecrawl -> Jina -> Crawl4AI -> simple).
Unsupported files used to enqueue a background job that failed and then
burned the full 15-attempt retry budget (~1h) before showing a generic
"Failed" with no actionable detail.
- Add a pre-flight `_assert_file_supported()` using content-core 2.x's
header-only `check_file_support()` (same routing as real extraction) in
the upload branch of `_build_content_state`, before any job is enqueued
and before the source record is created.
- Guard the source-retry endpoint the same way. Unexpected check errors
(e.g. file removed before a retry) fall through to normal extraction.
- Map `UnsupportedTypeException` to `415 Unsupported Media Type` via a
dedicated global handler (previously fell through to the base
OpenNotebookError handler's 500). The message names the detected type.
Part of #939.
* chore(sources): upgrade to content-core 2.x
Bump content-core 1.14.x -> 2.0.4 and adapt the source graph to the new
keyword-only extract_content API.
- extract_content is now keyword-only; engine/model overrides move to
ContentCoreConfig (audio model still sourced from Default Models).
- ProcessSourceState was removed; the graph now consumes ExtractionOutput,
which no longer echoes url/file_path back — carry those from the input
state into the saved Asset.
- content-core 2.x no longer deletes the uploaded source file, so honor
delete_source on our side after a successful extraction.
- drop the obsolete output_format param (markdown is the default).
pymupdf (AGPL) is replaced transitively by pdfplumber (MIT); moviepy is
gone (direct ffmpeg), which fixes MP3-with-chapters audio.
Part of #939.
* docs(changelog): note content-core 2.x upgrade (#1103)
* fix(sources): wire YouTube transcript language preferences into ContentCoreConfig
content-core's default youtube_languages is only en/es/pt. Pass the broader
list Open Notebook has always intended so non-English videos still resolve a
transcript. (cubic P2 on #1116)
Since #1112 dropped the legacy provider/model strings, EpisodeCard's outline,
transcript and speaker rows showed "— / —" for new episodes. The API now
resolves the snapshot's model references (outline_llm/transcript_llm/voice_model)
to provider/name display fields at serialization time, batched into a single
query per request via Model.get_display_info_for_ids so listing episodes never
does a per-row model lookup. The card falls back to the legacy snapshot strings
for old episodes and degrades to a dash when a referenced model was deleted.
Closes#1114
