From fcfd2afdb4f74fe6464e17cc90eebe1bc37a6e02 Mon Sep 17 00:00:00 2001 From: Luis Novo Date: Sun, 19 Jul 2026 18:11:00 -0300 Subject: [PATCH] fix(proxy): keep internal SurrealDB websocket out of HTTP proxy (#1185) * fix(proxy): keep internal SurrealDB websocket out of HTTP proxy (#1160) websockets 15.0 auto-detects HTTP_PROXY/HTTPS_PROXY and tunnels even ws:// connections through the proxy. The SurrealDB SDK connects over a websocket, so with a proxy set the internal DB connection was routed through the external proxy, which rejected the internal host with HTTP 403 and killed the worker/API on startup. - Add ensure_internal_no_proxy() helper that merges host.docker.internal, surrealdb, localhost, 127.0.0.1 into no_proxy/NO_PROXY (never clobbering a user value) and call it at API, worker and DB-module startup. - Add host.docker.internal and surrealdb to the .env.example and docs NO_PROXY examples. - Add unit tests for the injection helper. * fix(proxy): preserve NO_PROXY wildcard and include custom SurrealDB host (#1160) Address review findings on the no_proxy injection: - NO_PROXY=* (bypass all hosts) is now treated as terminal: leave the user's config untouched instead of narrowing the wildcard to a finite list by appending the internal hosts. - Parse the SurrealDB host from SURREAL_URL (falling back to SURREAL_ADDRESS) and add it to the bypass list, so deployments with a custom DB host/IP no longer route DB traffic through the proxy. Unset or malformed values fall back to the four defaults gracefully. - Drop the inaccurate getproxies() caching remark in the test. --- .env.example | 10 ++ CHANGELOG.md | 1 + api/main.py | 7 + commands/__init__.py | 8 + docs/5-CONFIGURATION/environment-reference.md | 10 +- open_notebook/database/repository.py | 7 + open_notebook/utils/proxy.py | 118 +++++++++++++++ tests/test_proxy.py | 138 ++++++++++++++++++ 8 files changed, 295 insertions(+), 4 deletions(-) create mode 100644 open_notebook/utils/proxy.py create mode 100644 tests/test_proxy.py diff --git a/.env.example b/.env.example index 30d8a65c..050ae567 100644 --- a/.env.example +++ b/.env.example @@ -74,5 +74,15 @@ SURREAL_DATABASE=open_notebook # BASIC_AUTH_USERNAME=admin # BASIC_AUTH_PASSWORD=secret +# Network / Proxy (corporate / firewalled environments) +# Route outbound HTTP(S) through a proxy. NO_PROXY MUST list the internal +# SurrealDB hosts (host.docker.internal, surrealdb) — otherwise the DB +# websocket gets tunnelled through the proxy and the worker/API fail to start +# with HTTP 403. The app also injects these hosts automatically as a safety +# net, but keep them here so your config is explicit. +# HTTP_PROXY=http://proxy.corp.com:8080 +# HTTPS_PROXY=http://proxy.corp.com:8080 +# NO_PROXY=localhost,127.0.0.1,host.docker.internal,surrealdb,.local + # For more configuration options, see: # https://github.com/lfnovo/open-notebook/blob/main/docs/5-CONFIGURATION/environment-reference.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 8512d79e..f09e3ce0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Four new AI providers** (surfaced by esperanto 2.25.1) are now in the provider matrix. **Cohere** (`COHERE_API_KEY`) adds language and embedding models via Cohere's native v2 API (`command-*` chat, `embed-*`); model discovery is bespoke (esperanto's `AIFactory.get_provider_models`) since Cohere is not OpenAI-compatible, and Cohere reranking is intentionally left out of scope (tracked at #1087). **PayPerQ / PPQ** (`PPQ_API_KEY`, `https://api.ppq.ai/v1`) is a multi-modality OpenAI-compatible gateway offering language, embedding, speech-to-text and text-to-speech. **Novita** (`NOVITA_API_KEY`, `https://api.novita.ai/openai`) is an OpenAI-compatible LLM gateway. PPQ and Novita auto-discover their models through the standard `/models` endpoint. In addition, the existing **Deepgram** provider now also supports **speech-to-text** (Nova/Whisper transcription models, e.g. `nova-3`) alongside its Aura text-to-speech voices. All four are configured from Settings → Models exactly like the existing providers (#1170) ### Fixed +- **HTTP proxy no longer breaks worker/API startup.** `websockets` 15.0 began auto-detecting `HTTP_PROXY`/`HTTPS_PROXY` and tunnels even `ws://` connections through the proxy, so with a proxy set the internal SurrealDB websocket (`ws://host.docker.internal:8018/rpc` or `ws://surrealdb:8000/rpc`) was routed through the external proxy, which rejected the internal host with HTTP 403 and killed the worker on startup. Open Notebook now injects `host.docker.internal,surrealdb,localhost,127.0.0.1` into `no_proxy`/`NO_PROXY` at startup (merged with any user value, never clobbering it), and the `.env.example` / docs `NO_PROXY` examples now include the internal DB hosts (#1160) - Auto-assign no longer silently re-populates optional model defaults that were deliberately cleared. Previously `POST /models/auto-assign` treated every empty slot as "missing" and filled it, so an optional slot a user intentionally cleared (to fall back to the chat model) got re-assigned on the next run. Auto-assign now fills only the two required slots (chat, embedding); the optional slots (transformation, tools, large context, TTS, STT) are left untouched. `large_context` now also falls back to the chat model when unset (matching transformation/tools) instead of returning nothing, and the Settings UI shows an inline hint on each empty optional slot — "using chat model (…)" for the text slots, "not configured" for TTS/STT (#1098) - PPQ model discovery now lists all modalities, not just chat. PPQ's `/v1/models` returns only chat/language models by default; the discovery URL now requests `?type=all` so the embedding, speech-to-text and text-to-speech models this multi-modality gateway advertises actually surface in the provider matrix (`PPQ_MODEL_TYPES` classifies them into the right slots) (#1180) - Vertex credentials now configure text-to-speech (and other Vertex providers) correctly when used through a stored credential rather than environment variables. `Credential.to_esperanto_config()` emitted the generic `project`/`location` keys, but esperanto's Vertex providers expect `vertex_project`/`vertex_location` — so credential-linked Vertex TTS crashed with `__init__() got an unexpected keyword argument 'project'`. The config now maps to the Vertex-specific key names for the `vertex` provider only; the `Credential` schema/API fields stay `project`/`location`, and non-Vertex providers are unaffected (#1151) diff --git a/api/main.py b/api/main.py index 2b8af38f..0a7c2db3 100644 --- a/api/main.py +++ b/api/main.py @@ -3,6 +3,13 @@ from dotenv import load_dotenv load_dotenv() +# Keep the internal SurrealDB websocket out of any configured HTTP proxy +# (issue #1160). Must run after load_dotenv() (so a proxy set in .env is +# already visible) and before the DB is touched. +from open_notebook.utils.proxy import ensure_internal_no_proxy + +ensure_internal_no_proxy() + import asyncio import os from contextlib import asynccontextmanager diff --git a/commands/__init__.py b/commands/__init__.py index fdda740e..bac4fb74 100644 --- a/commands/__init__.py +++ b/commands/__init__.py @@ -1,5 +1,13 @@ """Surreal-commands integration for Open Notebook""" +# The worker starts via `surreal-commands-worker --import-modules commands`, +# so this package is imported before the worker connects to SurrealDB. Inject +# the internal DB hosts into no_proxy first so the DB websocket is never +# tunnelled through a configured HTTP proxy (issue #1160). +from open_notebook.utils.proxy import ensure_internal_no_proxy + +ensure_internal_no_proxy() + from .embedding_commands import ( embed_insight_command, embed_note_command, diff --git a/docs/5-CONFIGURATION/environment-reference.md b/docs/5-CONFIGURATION/environment-reference.md index 3eee2f8d..e7a082b9 100644 --- a/docs/5-CONFIGURATION/environment-reference.md +++ b/docs/5-CONFIGURATION/environment-reference.md @@ -137,10 +137,12 @@ The `CCORE_FIRECRAWL_*` variables are passed straight through to the content-cor |----------|-----------|---------|-------------| | `HTTP_PROXY` | No | None | HTTP proxy URL for outbound HTTP requests | | `HTTPS_PROXY` | No | None | HTTPS proxy URL for outbound HTTPS requests | -| `NO_PROXY` | No | None | Comma-separated list of hosts to bypass proxy | +| `NO_PROXY` | No | None | Comma-separated list of hosts to bypass proxy (must include the internal DB hosts — see below) | Route all outbound HTTP requests through a proxy server. Useful for corporate/firewalled environments. +> **Important:** `NO_PROXY` must list the internal SurrealDB hosts — `host.docker.internal` (Docker) and `surrealdb` (the compose service name). The SurrealDB SDK connects over a websocket, and `websockets` 15.0+ tunnels even `ws://` connections through a configured proxy, which then rejects the internal host with **HTTP 403** and prevents the API and worker from starting. Open Notebook injects `host.docker.internal,surrealdb,localhost,127.0.0.1` into `NO_PROXY` automatically at startup as a safety net, but you should still set them explicitly. + The underlying libraries (esperanto, content-core, podcast-creator) automatically detect proxy settings from these standard environment variables. **Affects:** @@ -160,8 +162,8 @@ HTTPS_PROXY=http://proxy.corp.com:8080 HTTP_PROXY=http://user:password@proxy.corp.com:8080 HTTPS_PROXY=http://user:password@proxy.corp.com:8080 -# Bypass proxy for local hosts -NO_PROXY=localhost,127.0.0.1,.local +# Bypass proxy for local hosts (include the internal DB hosts!) +NO_PROXY=localhost,127.0.0.1,host.docker.internal,surrealdb,.local ``` --- @@ -212,7 +214,7 @@ API_URL=https://mynotebook.example.com OPEN_NOTEBOOK_ENCRYPTION_KEY=your-secret-key HTTP_PROXY=http://proxy.corp.com:8080 HTTPS_PROXY=http://proxy.corp.com:8080 -NO_PROXY=localhost,127.0.0.1 +NO_PROXY=localhost,127.0.0.1,host.docker.internal,surrealdb,.local ``` ### High-Performance Deployment diff --git a/open_notebook/database/repository.py b/open_notebook/database/repository.py index 6a760de8..7eb960b6 100644 --- a/open_notebook/database/repository.py +++ b/open_notebook/database/repository.py @@ -8,6 +8,13 @@ from loguru import logger from surrealdb import AsyncSurreal, RecordID # type: ignore from surrealdb.data.types.table import Table # type: ignore +from open_notebook.utils.proxy import ensure_internal_no_proxy + +# Keep the internal SurrealDB websocket out of any configured HTTP proxy +# (issue #1160). Runs at import time - i.e. before any db_connection() can be +# opened - so it protects every entrypoint (API + worker) that touches the DB. +ensure_internal_no_proxy() + T = TypeVar("T", Dict[str, Any], List[Dict[str, Any]]) # Bare SurrealDB table/relation identifier: no ':', whitespace, or query diff --git a/open_notebook/utils/proxy.py b/open_notebook/utils/proxy.py new file mode 100644 index 00000000..ab853c55 --- /dev/null +++ b/open_notebook/utils/proxy.py @@ -0,0 +1,118 @@ +"""Proxy environment helpers. + +Defensive handling for HTTP proxy setups (issue #1160). + +``websockets`` 15.0 started auto-detecting ``HTTP_PROXY`` / ``HTTPS_PROXY`` +from the environment and tunnels *every* connection through the proxy, +including plain ``ws://`` ones. The SurrealDB SDK connects to the database +over a websocket, so when a proxy is configured the *internal* DB connection +(e.g. ``ws://host.docker.internal:8018/rpc`` or ``ws://surrealdb:8000/rpc``) +gets routed through the external proxy, which rejects the internal host with +HTTP 403 and kills the API / worker on startup. + +To make this robust regardless of user configuration, we inject the internal +DB hosts into the ``no_proxy`` / ``NO_PROXY`` env vars at startup (merged with +any user-provided value, never clobbering it). ``urllib.request.proxy_bypass`` +- which ``websockets`` calls to decide whether to tunnel - reads these vars, +so the internal websocket is left un-proxied. +""" + +import os +from urllib.parse import urlsplit + +# Hosts the DB is reachable at internally. These must never be routed through +# an external proxy. Covers Docker (host.docker.internal, surrealdb service +# name) and local (localhost / 127.0.0.1) topologies. Deployments that point +# at a custom SurrealDB host/IP have it added dynamically - see +# _configured_db_host(). +INTERNAL_NO_PROXY_HOSTS = ( + "host.docker.internal", + "surrealdb", + "localhost", + "127.0.0.1", +) + +# urllib.request.proxy_bypass_environment reads the lowercase var first, then +# falls back to the uppercase one. We keep both in sync so the internal hosts +# are honored no matter which variant a user set. +_NO_PROXY_ENV_VARS = ("no_proxy", "NO_PROXY") + + +def _split_hosts(value: str) -> list[str]: + return [h.strip() for h in value.split(",") if h.strip()] + + +def _configured_db_host() -> str | None: + """Best-effort extraction of the SurrealDB host from the environment. + + Mirrors the resolution order in ``open_notebook.database.repository`` - + ``SURREAL_URL`` wins, otherwise the legacy ``SURREAL_ADDRESS``. Returns the + bare host (no scheme/port/path), or ``None`` when unset or unparseable. + + Parsed here from the env directly (rather than importing repository) to + avoid a circular import - repository imports this module at load time. + """ + candidate = os.environ.get("SURREAL_URL", "").strip() + if not candidate: + candidate = os.environ.get("SURREAL_ADDRESS", "").strip() + if not candidate: + return None + + # urlsplit only populates .hostname when a scheme is present; add a dummy + # one for bare ``host`` / ``host:port`` values (e.g. SURREAL_ADDRESS). + if "://" not in candidate: + candidate = "ws://" + candidate + + try: + host = urlsplit(candidate).hostname + except ValueError: + return None + return host or None + + +def _internal_hosts() -> list[str]: + """The default internal hosts plus the configured SurrealDB host, if any.""" + hosts = list(INTERNAL_NO_PROXY_HOSTS) + db_host = _configured_db_host() + if db_host and db_host.lower() not in {h.lower() for h in hosts}: + hosts.append(db_host) + return hosts + + +def ensure_internal_no_proxy() -> None: + """Ensure internal DB hosts bypass any configured proxy. + + Merges the internal hosts (see :func:`_internal_hosts`) into both + ``no_proxy`` and ``NO_PROXY`` (preserving existing entries and their order) + and writes the combined value back to both env vars. Idempotent - safe to + call more than once and from multiple entrypoints (API + worker). + + A wildcard (``*``) already bypasses the proxy for every host, so if the + user set ``no_proxy``/``NO_PROXY`` to (or containing) a bare ``*`` we leave + their configuration untouched rather than turning it into a finite list. + """ + # Collect existing entries from whichever variants the user set, de-duped + # while preserving order. + existing: list[str] = [] + seen: set[str] = set() + for var in _NO_PROXY_ENV_VARS: + for host in _split_hosts(os.environ.get(var, "")): + # Wildcard is terminal: it already bypasses everything (including + # the internal DB hosts), so don't narrow it to a finite list. + if host == "*": + return + key = host.lower() + if key not in seen: + seen.add(key) + existing.append(host) + + # Append internal hosts that aren't already present. + merged = list(existing) + for host in _internal_hosts(): + if host.lower() not in seen: + seen.add(host.lower()) + merged.append(host) + + combined = ",".join(merged) + for var in _NO_PROXY_ENV_VARS: + os.environ[var] = combined diff --git a/tests/test_proxy.py b/tests/test_proxy.py new file mode 100644 index 00000000..da8f128c --- /dev/null +++ b/tests/test_proxy.py @@ -0,0 +1,138 @@ +"""Tests for the internal no_proxy injection (issue #1160).""" + +import os + +import pytest + +from open_notebook.utils.proxy import ( + INTERNAL_NO_PROXY_HOSTS, + ensure_internal_no_proxy, +) + +_PROXY_VARS = ("no_proxy", "NO_PROXY") +_DB_VARS = ("SURREAL_URL", "SURREAL_ADDRESS") + + +@pytest.fixture(autouse=True) +def _clean_proxy_env(monkeypatch): + """Ensure each test starts with the proxy / DB-host vars unset.""" + for var in (*_PROXY_VARS, *_DB_VARS): + monkeypatch.delenv(var, raising=False) + yield + + +def test_injects_internal_hosts_when_unset(): + ensure_internal_no_proxy() + for var in _PROXY_VARS: + value = os.environ[var] + for host in INTERNAL_NO_PROXY_HOSTS: + assert host in value.split(",") + + +def test_preserves_user_value(monkeypatch): + monkeypatch.setenv("NO_PROXY", "example.com,10.0.0.5") + ensure_internal_no_proxy() + entries = os.environ["NO_PROXY"].split(",") + # User entries kept, in their original order and at the front. + assert entries[:2] == ["example.com", "10.0.0.5"] + # Internal hosts appended. + for host in INTERNAL_NO_PROXY_HOSTS: + assert host in entries + + +def test_does_not_duplicate_existing_hosts(monkeypatch): + monkeypatch.setenv("NO_PROXY", "surrealdb,example.com") + ensure_internal_no_proxy() + entries = os.environ["NO_PROXY"].split(",") + assert entries.count("surrealdb") == 1 + + +def test_lowercase_and_uppercase_kept_in_sync(monkeypatch): + monkeypatch.setenv("no_proxy", "example.com") + ensure_internal_no_proxy() + assert os.environ["no_proxy"] == os.environ["NO_PROXY"] + + +def test_merges_both_case_variants(monkeypatch): + monkeypatch.setenv("no_proxy", "lower.example.com") + monkeypatch.setenv("NO_PROXY", "UPPER.example.com") + ensure_internal_no_proxy() + combined = os.environ["no_proxy"] + assert "lower.example.com" in combined + assert "UPPER.example.com" in combined + # De-duped case-insensitively, so no crash and both variants match. + assert os.environ["no_proxy"] == os.environ["NO_PROXY"] + + +def test_idempotent(monkeypatch): + monkeypatch.setenv("NO_PROXY", "example.com") + ensure_internal_no_proxy() + first = os.environ["NO_PROXY"] + ensure_internal_no_proxy() + assert os.environ["NO_PROXY"] == first + + +def test_wildcard_preserved(monkeypatch): + """NO_PROXY=* bypasses every host; it must not be narrowed to a list.""" + monkeypatch.setenv("NO_PROXY", "*") + ensure_internal_no_proxy() + assert os.environ["NO_PROXY"] == "*" + + +def test_wildcard_among_entries_preserved(monkeypatch): + """A bare * among other entries is still terminal - leave config as-is.""" + monkeypatch.setenv("NO_PROXY", "example.com,*") + ensure_internal_no_proxy() + assert os.environ["NO_PROXY"] == "example.com,*" + + +def test_custom_surreal_url_host_included(monkeypatch): + monkeypatch.setenv("SURREAL_URL", "ws://db.internal.corp:8000/rpc") + ensure_internal_no_proxy() + entries = os.environ["NO_PROXY"].split(",") + assert "db.internal.corp" in entries + # Defaults still present. + for host in INTERNAL_NO_PROXY_HOSTS: + assert host in entries + + +def test_custom_surreal_address_host_included(monkeypatch): + # Legacy address form (host:port, no scheme). + monkeypatch.setenv("SURREAL_ADDRESS", "10.1.2.3:8000") + ensure_internal_no_proxy() + assert "10.1.2.3" in os.environ["NO_PROXY"].split(",") + + +def test_surreal_url_takes_precedence_over_address(monkeypatch): + monkeypatch.setenv("SURREAL_URL", "ws://from-url:8000/rpc") + monkeypatch.setenv("SURREAL_ADDRESS", "from-address") + ensure_internal_no_proxy() + entries = os.environ["NO_PROXY"].split(",") + assert "from-url" in entries + assert "from-address" not in entries + + +def test_malformed_surreal_url_falls_back_to_defaults(monkeypatch): + monkeypatch.setenv("SURREAL_URL", "::not a url::") + ensure_internal_no_proxy() + entries = os.environ["NO_PROXY"].split(",") + # No crash; the four defaults are still injected. + for host in INTERNAL_NO_PROXY_HOSTS: + assert host in entries + + +def test_custom_host_not_duplicated_when_default(monkeypatch): + monkeypatch.setenv("SURREAL_URL", "ws://surrealdb:8000/rpc") + ensure_internal_no_proxy() + assert os.environ["NO_PROXY"].split(",").count("surrealdb") == 1 + + +def test_bypass_recognized_by_urllib(monkeypatch): + """The injected hosts are actually honored by urllib.request.proxy_bypass, + which is what websockets calls to decide whether to tunnel.""" + import urllib.request + + monkeypatch.setenv("HTTP_PROXY", "http://proxy.corp.com:8080") + ensure_internal_no_proxy() + assert urllib.request.proxy_bypass("surrealdb:8000") + assert urllib.request.proxy_bypass("host.docker.internal:8018")