mirror of
https://github.com/lfnovo/open-notebook.git
synced 2026-04-28 11:30:00 +00:00
d0bbe4a921
In air-gapped / offline Docker deployments, tiktoken.get_encoding() tries
to download the encoding file from openaipublic.blob.core.windows.net.
When that request fails it raises a URLError / OSError — not an ImportError
— so the previous except clause silently missed it and the crash surfaced in
the UI.
Widened `except ImportError` to `except Exception` so all failures —
"not installed" and "network unreachable" — fall through to the word-count
fallback (words × 1.3). Added a loguru WARNING so operators can see when
the fallback is active.
TIKTOKEN_CACHE_DIR now reads from the environment with a blank-safe
fallback (`or` guard prevents os.makedirs("") on empty env var). This lets
Docker images redirect the cache to a path outside /app/data/ so user-data
volume mounts cannot shadow the pre-baked encoding.
Both images now pre-download the o200k_base encoding during the builder
stage (internet is available at build time) and copy it into the runtime
image at /app/tiktoken-cache. ENV TIKTOKEN_CACHE_DIR=/app/tiktoken-cache
is set in the runtime stage so no network call is ever needed at runtime.
Added test_token_count_network_error_fallback in tests/test_utils.py:
patches tiktoken.get_encoding with a URLError and asserts token_count()
returns a positive int instead of raising.
Fixes #264
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
20 lines
733 B
Python
20 lines
733 B
Python
import os
|
|
|
|
# ROOT DATA FOLDER
|
|
DATA_FOLDER = "./data"
|
|
|
|
# LANGGRAPH CHECKPOINT FILE
|
|
sqlite_folder = f"{DATA_FOLDER}/sqlite-db"
|
|
os.makedirs(sqlite_folder, exist_ok=True)
|
|
LANGGRAPH_CHECKPOINT_FILE = f"{sqlite_folder}/checkpoints.sqlite"
|
|
|
|
# UPLOADS FOLDER
|
|
UPLOADS_FOLDER = f"{DATA_FOLDER}/uploads"
|
|
os.makedirs(UPLOADS_FOLDER, exist_ok=True)
|
|
|
|
# TIKTOKEN CACHE FOLDER
|
|
# Reads TIKTOKEN_CACHE_DIR from the environment so Docker can redirect the cache
|
|
# to a path outside /data/ (which is typically volume-mounted and would hide the
|
|
# pre-baked encoding baked into the image at build time).
|
|
TIKTOKEN_CACHE_DIR = os.environ.get("TIKTOKEN_CACHE_DIR", "").strip() or f"{DATA_FOLDER}/tiktoken-cache"
|
|
os.makedirs(TIKTOKEN_CACHE_DIR, exist_ok=True)
|