open-notebook/api
Luis Novo ef5bd2ce7b
feat(sources): add Crawl4AI URL engine + honor persisted engine settings (#432) (#1118)
* feat(sources): add Crawl4AI URL engine and honor persisted engine settings (#432)

- Add "crawl4ai" as a selectable URL processing engine (domain Literal,
  settings API validation, SettingsForm select, and label across all 14
  locales; urlHelp updated in en-US to describe the new fallback chain).
- The source graph now loads the persisted ContentSettings and passes the
  document/URL engine choices to ContentCoreConfig. Previously it built a
  hard-coded ContentSettings with "auto" engines, so a user's selection in
  Settings never took effect. Falls back to defaults if settings can't load.
- Crawl4AI Docker mode is driven by content-core's native CRAWL4AI_API_URL
  env var (documented separately under #1105).

Part of #939.

* fix(432): bundle Crawl4AI runtime + address review

- Bundle the Crawl4AI runtime so its local, no-API-key mode works out of
  the box: depend on content-core[crawl4ai] and install the Chromium
  browser via playwright in the Docker runtime-base (both image variants).
  Footprint is modest (no torch/transformers/CUDA); image grows ~300 MB
  from Chromium + system libs.
- Preserve the server-side traceback when persisted content settings fail
  to load (logger.opt(exception=True)) instead of only the message.
- Reset the ContentSettings singleton between domain tests (clear_instance)
  so a non-default value can't leak into neighboring tests.

Addresses review on #432.

* i18n(432): translate urlHelp Crawl4AI description across all 13 non-en locales

The Crawl4AI engine label was already localized; this brings the URL-engine
'help me choose' text in line with en-US in every locale — describing
Crawl4AI (local JS rendering, no API key) and its place in the auto
fallback chain (Firecrawl -> Jina -> Crawl4AI -> simple).
2026-07-12 20:05:03 -03:00
..
routers feat(sources): add Crawl4AI URL engine + honor persisted engine settings (#432) (#1118) 2026-07-12 20:05:03 -03:00
__init__.py Api podcast migration (#93) 2025-07-17 08:36:11 -03:00
AGENTS.md docs: restructure documentation around AGENTS.md, VISION.md and decision records (#1032) 2026-07-10 15:33:19 -03:00
auth.py chore: remove dead auth helper and fix stale default-password docs (#1026) 2026-07-10 15:47:40 -03:00
CLAUDE.md docs: restructure documentation around AGENTS.md, VISION.md and decision records (#1032) 2026-07-10 15:33:19 -03:00
command_service.py refactor(api): remove dead Streamlit-era service layer and demo commands (#1054) 2026-07-11 18:17:44 -03:00
credentials_service.py ci: gate PRs on mypy, start ignore_errors burn-down (#1076) 2026-07-11 19:25:29 -03:00
main.py fix(sources): reject unsupported uploads at ingestion with 415 (#975) (#1117) 2026-07-12 18:59:59 -03:00
middleware.py fix: reject oversized request bodies before auth/routing (#1014) 2026-07-10 11:15:18 -03:00
models.py refactor(frontend): consume GET /api/providers instead of the manual provider table (#1108) 2026-07-12 12:46:38 -03:00
podcast_service.py fix(podcasts): reference speaker profiles by record ID instead of name (#1110) 2026-07-12 13:05:04 -03:00