vrr/open-notebook
2
0
Fork 0
mirror of https://github.com/lfnovo/open-notebook.git synced 2026-04-28 19:40:50 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix: narrow exception to (ImportError, OSError) and include error in log

Browse source 
Broad 'except Exception' could silently swallow unexpected failures.
URLError and ConnectionError are both subclasses of OSError, so
'except (ImportError, OSError)' captures all real offline/not-installed
cases while letting genuine programming errors propagate.

Also include the exception detail in the warning message so failures
are diagnosable in logs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
orihatav 2026-02-21 22:59:55 +02:00 • committed by Luis Novo
parent d0bbe4a921
commit 5a350a7622
1 changed files with 5 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

10
open_notebook/utils/token_utils.py
View file

@ -28,14 +28,14 @@ def token_count(input_string: str) -> int:
encoding = tiktoken.get_encoding("o200k_base")
tokens = encoding.encode(input_string)
return len(tokens)
except Exception:
# Fallback: handles ImportError (not installed) AND network errors
# (e.g., offline environments that can't download encoding from internet)
except (ImportError, OSError) as e:
# Fallback: handles ImportError (tiktoken not installed) AND network/OS
# errors such as urllib.error.URLError or ConnectionError raised in
# offline environments when the encoding file cannot be downloaded.
from loguru import logger
logger.warning(
"tiktoken unavailable (not installed or offline); "
"falling back to word-count estimation."
"tiktoken unavailable, falling back to word-count estimation: {}", e
)
return int(len(input_string.split()) * 1.3)