open-notebook/tests
Luis Novo ef5bd2ce7b
feat(sources): add Crawl4AI URL engine + honor persisted engine settings (#432) (#1118)
* feat(sources): add Crawl4AI URL engine and honor persisted engine settings (#432)

- Add "crawl4ai" as a selectable URL processing engine (domain Literal,
  settings API validation, SettingsForm select, and label across all 14
  locales; urlHelp updated in en-US to describe the new fallback chain).
- The source graph now loads the persisted ContentSettings and passes the
  document/URL engine choices to ContentCoreConfig. Previously it built a
  hard-coded ContentSettings with "auto" engines, so a user's selection in
  Settings never took effect. Falls back to defaults if settings can't load.
- Crawl4AI Docker mode is driven by content-core's native CRAWL4AI_API_URL
  env var (documented separately under #1105).

Part of #939.

* fix(432): bundle Crawl4AI runtime + address review

- Bundle the Crawl4AI runtime so its local, no-API-key mode works out of
  the box: depend on content-core[crawl4ai] and install the Chromium
  browser via playwright in the Docker runtime-base (both image variants).
  Footprint is modest (no torch/transformers/CUDA); image grows ~300 MB
  from Chromium + system libs.
- Preserve the server-side traceback when persisted content settings fail
  to load (logger.opt(exception=True)) instead of only the message.
- Reset the ContentSettings singleton between domain tests (clear_instance)
  so a non-default value can't leak into neighboring tests.

Addresses review on #432.

* i18n(432): translate urlHelp Crawl4AI description across all 13 non-en locales

The Crawl4AI engine label was already localized; this brings the URL-engine
'help me choose' text in line with en-US in every locale — describing
Crawl4AI (local JS rendering, no API key) and its place in the auto
fallback chain (Firecrawl -> Jina -> Crawl4AI -> simple).
2026-07-12 20:05:03 -03:00
..
conftest.py feat: credential-based API key management (#477) (#540) 2026-02-10 08:30:22 -03:00
README.md Initial commit with all features 2024-10-21 14:56:10 -03:00
test_add_insight_failure_propagation.py ci: gate PRs on mypy, start ignore_errors burn-down (#1076) 2026-07-11 19:25:29 -03:00
test_chat_routers_characterization.py fix(api): let typed domain exceptions reach the global handlers (#1078) 2026-07-11 19:42:10 -03:00
test_chunking.py fix(embedding): drop degenerate tiny chunks before embedding (#764) (#768) 2026-05-31 09:43:47 -03:00
test_config_endpoint_no_leak.py fix: don't leak internal exception text in API error responses (#1017) 2026-07-10 13:40:27 -03:00
test_connection_test_classification.py fix: make the provider connection test resilient to model retirement (#1035) 2026-07-10 16:26:22 -03:00
test_context_endpoint_characterization.py refactor(api): single context-building implementation (#1079) 2026-07-11 19:55:46 -03:00
test_cors_credentials.py ci: gate PRs on mypy, start ignore_errors burn-down (#1076) 2026-07-11 19:25:29 -03:00
test_credential_provider_validation.py refactor(frontend): consume GET /api/providers instead of the manual provider table (#1108) 2026-07-12 12:46:38 -03:00
test_credentials_api.py fix: actually clear credential fields end to end (frontend payload + API null handling) (#1046) 2026-07-11 09:00:06 -03:00
test_crud_404.py fix(api): return 404 instead of 500 for missing resources in CRUD endpoints (#862) (#924) 2026-06-18 08:23:28 -03:00
test_domain.py feat(sources): add Crawl4AI URL engine + honor persisted engine settings (#432) (#1118) 2026-07-12 20:05:03 -03:00
test_embedding.py chore(lint): re-enable F401/F841/E722 and fix fallout (#1062) 2026-07-11 18:36:04 -03:00
test_error_message_sanitization.py fix: don't leak internal exception text in API error responses (#1017) 2026-07-10 13:40:27 -03:00
test_graphs.py feat(sources): add Crawl4AI URL engine + honor persisted engine settings (#432) (#1118) 2026-07-12 20:05:03 -03:00
test_insight_timestamps.py fix(insights): stamp created/updated on source_insight and stop serializing "None" (#1085) 2026-07-12 11:05:51 -03:00
test_max_body_size_middleware.py ci: gate PRs on mypy, start ignore_errors burn-down (#1076) 2026-07-11 19:25:29 -03:00
test_model_discovery.py refactor(ai): table-driven model discovery, real Anthropic listing (#1070) 2026-07-11 18:54:46 -03:00
test_models_api.py fix(models): allow clearing optional model defaults (#1097) 2026-07-12 09:47:17 -03:00
test_note_save_embed_resilience.py fix: surface silent command-submission failures where they matter (#1019) 2026-07-10 13:53:57 -03:00
test_notes_api.py feat: expose embed command_id in note API responses (#545) 2026-02-14 18:11:23 -03:00
test_order_by_validation.py fix: prevent SurrealQL injection in Credential.get_all()'s order_by (#1021) 2026-07-10 14:10:01 -03:00
test_podcast_audio_containment.py refactor(podcasts): store audio paths relative to PODCASTS_FOLDER (#1111) 2026-07-12 14:00:34 -03:00
test_podcast_audio_paths.py refactor(podcasts): store audio paths relative to PODCASTS_FOLDER (#1111) 2026-07-12 14:00:34 -03:00
test_podcast_episode_model_resolution.py fix(podcasts): render EpisodeCard model details from resolved model references (#1115) 2026-07-12 14:57:54 -03:00
test_podcast_job_status_batching.py ci: gate PRs on mypy, start ignore_errors burn-down (#1076) 2026-07-11 19:25:29 -03:00
test_podcast_legacy_field_removal.py chore(podcasts): drop legacy provider/model string fields from podcast profiles (#1107) (#1112) 2026-07-12 14:35:04 -03:00
test_podcast_path.py refactor(podcasts): store audio paths relative to PODCASTS_FOLDER (#1111) 2026-07-12 14:00:34 -03:00
test_podcast_speaker_profile.py refactor(podcasts): store audio paths relative to PODCASTS_FOLDER (#1111) 2026-07-12 14:00:34 -03:00
test_recently_viewed_api.py feat: Recently Viewed section for notebooks and sources (#979) 2026-07-03 12:39:14 -03:00
test_repository_config.py refactor(types): type-check domain base model (#961) 2026-06-25 08:16:41 -03:00
test_search_api.py fix(search): reject non-positive limit and survive highlight overflow (#898) 2026-06-16 04:51:34 -03:00
test_source_create_array_limits.py ci: gate PRs on mypy, start ignore_errors burn-down (#1076) 2026-07-11 19:25:29 -03:00
test_source_path_containment.py fix: harden source upload path handling and cap array inputs (#1015) 2026-07-10 11:32:51 -03:00
test_sources_api.py fix: sort sources by title without tripping the SEARCH index, return 422 for invalid form data (#1042) 2026-07-10 21:10:24 -03:00
test_startup_migration_retry.py ci: gate PRs on mypy, start ignore_errors burn-down (#1076) 2026-07-11 19:25:29 -03:00
test_transformations_api.py feat: per-transformation custom model selection (#978) 2026-07-03 12:30:50 -03:00
test_typed_exceptions_reach_handlers.py fix(sources): reject unsupported uploads at ingestion with 415 (#975) (#1117) 2026-07-12 18:59:59 -03:00
test_upload_toctou_race.py fix: harden source upload path handling and cap array inputs (#1015) 2026-07-10 11:32:51 -03:00
test_upload_type_mitigations.py fix(sources): reject unsupported uploads at ingestion with 415 (#975) (#1117) 2026-07-12 18:59:59 -03:00
test_url_validation.py fix: make validate_url() async so DNS resolution doesn't block the event loop (#1011) 2026-07-10 10:43:36 -03:00
test_utils.py refactor(api): single context-building implementation (#1079) 2026-07-11 19:55:46 -03:00
test_vertex_credentials_file_oracle.py fix: don't leak filesystem info via Vertex credential test errors (#1012) 2026-07-10 11:07:04 -03:00

Coming Soon