vrr/open-notebook
2
0
Fork 0
mirror of https://github.com/lfnovo/open-notebook.git synced 2026-04-29 03:50:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Feat/localization tests docker (#371)

Browse source 
* feat(i18n): complete 100% internationalization and fix Next.js 15 compatibility

* feat(i18n): complete 100% internationalization coverage

* chore(test): finalize component tests and project cleanup

* test(logic): add unit tests for useModalManager hook

* fix(test): resolve timeout in AppSidebar tests by mocking TooltipProvider

* feat(i18n): comprehensive i18n audit, fixes for hardcoded strings, and complete zh-TW support

* fix(i18n): resolve TypeScript warnings and improve translation hook stability

- Remove unused useTranslation import from ConnectionGuard
- Add ref-based checking state to prevent dependency cycles
- Fix useTranslation hook to return empty string for undefined translations
- Add comment for backward compatibility on ExtractedReference interface
- Ensure .replace() string methods work safely with nested translation keys

* feat(i18n): complete internationalization implementation with Docker deployment

- Add LanguageLoadingOverlay component for smooth language transitions
- Update all translation files (en-US, zh-CN, zh-TW) with improved terminology
- Optimize Docker configuration for better performance
- Update version check and config handling for i18n support
- Fix route handling for language-specific content
- Add comprehensive task documentation

* fix(i18n): resolve localization errors, duplicates, and type issues

* chore(i18n): finalize 100% internationalization coverage

* chore(test): supplement i18n test cases and cleanup redundant files

* fix(test): resolve lint type errors and finalize delivery documents

* feat(i18n): finalize full internationalization and zh-TW localization

* fix(frontend): add missing devDependency and fix build tsconfig

* feat(ui): enhance sidebar hover effects with better visual feedback

* fix(frontend): resolve accessibility, i18n, and lint issues

- fix: add missing id, name, autocomplete attributes to dialog inputs
- fix: add aria labels and DialogDescription for accessibility
- fix: resolve uncontrolled component warning in SettingsForm
- fix: correct duplicate 'Traditional Chinese' label in zh-TW locale
- feat: add i18n support for podcast template names
- chore: fix lint errors in Dialogs

* fix: address all 21 PR feedback items from cubic-dev-ai bot

Configuration:
- Remove ignoreDuringBuilds flags from next.config.ts

Testing:
- Fix AppSidebar.test.tsx regex pattern and add missing assertion

Logic:
- Fix ConnectionGuard.tsx re-entry prevention logic

Internationalization (I18n) - Translations:
- Add missing keys: notebooks.archived, common.note/insight, accessibility keys
- Add specific keys: sources.allSourcesDescShort, transformations.selectModel
- Add singular/plural keys: podcasts.usedByCount_one/other, common.note/notes
- Add common.created/updated with {time} placeholder

Internationalization (I18n) - Usage:
- SourcesPage: use allSourcesDescShort instead of string splitting
- TransformationPlayground: use navigation.transformation and selectModel
- CommandPalette: use dedicated keys instead of string concatenation
- GeneratePodcastDialog: fix zh-TW date locale handling
- NotebookHeader: correctly interpolate {time} placeholder
- TransformationCard: use common.description instead of undefined key
- ChatPanel/SpeakerProfilesPanel: implement proper pluralization
- SystemInfo: correctly interpolate {version} placeholder
- LanguageLoadingOverlay: use t.common.loading instead of hardcoded string
- MessageActions: use specific error key cannotSaveNoteNoNotebook

Other:
- Fix SessionManager.tsx exhaustive-deps warning

* fix: remove duplicate locale keys and add missing zh-CN translations

- en-US: remove duplicate loading key (line 59) and addNew key (sources)
- zh-CN: remove duplicate common keys (loading, note, insight, newSource, newNotebook, newPodcast)
- zh-CN: remove duplicate accessibility.searchNotebooks key
- zh-CN: remove duplicate sources.addNew key
- zh-CN: remove duplicate navigation.transformation key
- zh-CN: add missing usedByCount_one and usedByCount_other keys in podcasts
- zh-TW: remove duplicate common keys (loading, note, insight, newSource, newNotebook, newPodcast)
- zh-TW: remove duplicate accessibility.searchNotebooks key
- zh-TW: remove duplicate sources.addNew key

* docs: remove info.md

* fix: remove duplicate notebook keys and unused ts-expect-error

- zh-CN: remove duplicate notebooks keys (archived, archive, unarchive, deleteNotebook, deleteNotebookDesc)
- zh-TW: remove duplicate notebooks keys (archived, archive, unarchive, deleteNotebook, deleteNotebookDesc)
- GeneratePodcastDialog: remove unused @ts-expect-error directive

* fix(a11y): fix unassociated labels in search page

- Replace <Label> with role='group' + aria-labelledby for search type section
- Replace <Label> with role='group' + aria-labelledby for search in section
- Follows WAI-ARIA best practices for labeling form field groups

* fix(a11y): fix unassociated labels across multiple components

- search/page.tsx: use role='group' + aria-labelledby for search type and search in sections
- RebuildEmbeddings.tsx: use role='group' + aria-labelledby for include checkboxes
- TransformationPlayground.tsx: replace Label with span for non-form output label

* chore: revert to npm stack and ensure i18n compatibility

* chore: polish zh-TW translations for better idiomatic usage

* fix: resolve linter errors (ruff import sort, mypy config duplicate)

* style: apply ruff formatting

* fix: finalize upstream compliance (Dockerfile.single, i18n hooks, docker-compose)

* style: polish strings, fix timeout cleanup, and improve test mocks

* fix: use relative imports in test setup to resolve IDE path errors

* perf(docker): optimize build speed by removing apt-get upgrade and build tools

- Remove apt-get upgrade from both builder and runtime stages (saves 10-15 min each)
- Remove gcc/g++/make/git from builder (uv downloads pre-built wheels)
- Add --no-install-recommends to minimize package footprint
- Keep npm mirror (npmmirror.com) for faster frontend deps
- Add npm registry config for reliable China network access

Also includes:
- fix(a11y): add missing labels and aria attributes to form fields
- fix(i18n): add 2s safety timeout to LanguageLoadingOverlay
- fix(i18n): add robustness checks to use-translation proxy

Build time reduced from 2+ hours to ~34 minutes (~70% improvement)

* fix(a11y): resolve 16 form field accessibility warnings in notebook and podcast pages

* fix(a11y): resolve 4 button and 1 select field accessibility warnings in models page

* fix(a11y): resolve redundant attributes and residual warnings in transformations and podcast forms

* fix(i18n): deep fix for language switch hang using proxy protection and safer access

* fix(a11y): add name attributes to ModelSelector, TransformationPlayground, and SourceDetailContent

* fix: add missing Label import to SourceDetailContent

* fix(i18n): use native react-i18next in LanguageLoadingOverlay to prevent hang during language switch

* fix(i18n): rewrite use-translation Proxy with strict depth limit and expanded blocked props to prevent language switch hang

* fix: add type assertion to fix TypeScript comparison error

* fix(i18n): disable useSuspense to prevent thread hang during language resource loading

* fix(i18n): add infinite loop detection circuit breaker to useTranslation hook

* fix(i18n): update traditional chinese label to native script in en-US

* feat: add new localization strings for notebook and note management.

* fix: resolve config priority, docker build deps, and ui glitches

* refactor: improve ui details and test coverage based on feedback

* refactor: improve ui details (version check/lang toggle) and test coverage

* fix: polish language matching and test cleanup

* fix(test): update mocks to resolve timeouts and proxy errors

* fix(frontend): restore tsconfig.json structure and enable IDE support for tests

* fix: address PR review findings and resolve CI OIDC failure

* fix: merge exception headers in custom handler

* fix: comprehensive PR review remediations and async performance fixes

* refactor: address all PR #371 review feedback

- Docker: consolidate SURREAL_URL to docker.env, add single-container override
- Security: restore apt-get upgrade in Dockerfile and Dockerfile.single
- Create centralized getDateLocale helper (lib/utils/date-locale.ts)
- Refactor 7 files to use getDateLocale helper
- Revert config/route.ts to origin/main version
- Move test files to co-located pattern (3 files)
- Remove local useTranslation mock from ConfirmDialog.test.tsx
- Simplify use-version-check to single useEffect pattern
- Fix test import paths after moving to co-located pattern

* fix: add jest-dom types for test files

* fix: address remaining review issues

- Add apt-get upgrade -y to Dockerfile.single backend-builder stage
- Refactor ChatColumn.test.tsx: use 'as unknown as ReturnType<typeof hook>' instead of 'as any'
- Use toBeInTheDocument() assertions instead of toBeDefined()
This commit is contained in:
MisonL 2026-01-16 00:51:05 +08:00 • committed by GitHub
parent 940c56ddaf
commit 67dd85c928
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
179 changed files with 10201 additions and 2633 deletions
Download patch file Download diff file Expand all files Collapse all files

233
open_notebook/utils/context_builder.py
View file

@ -4,6 +4,7 @@ Generic ContextBuilder for the Open Notebook project.
This module provides a flexible ContextBuilder class that can handle any parameters
and build context from sources, notebooks, insights, and notes.
"""
from __future__ import annotations
from dataclasses import dataclass
@ -20,13 +21,13 @@ from .text_utils import token_count
@dataclass
class ContextItem:
"""Represents a single item in the context."""
id: str
type: Literal["source", "note", "insight"]
content: Dict[str, Any]
priority: int = 0
token_count: Optional[int] = None
def __post_init__(self):
"""Calculate token count for the content if not provided."""
if self.token_count is None:
@ -39,12 +40,12 @@ class ContextConfig:
"""Configuration for context building."""
sources: Optional[Dict[str, str]] = None # {source_id: inclusion_level}
notes: Optional[Dict[str, str]] = None # {note_id: inclusion_level}
notes: Optional[Dict[str, str]] = None # {note_id: inclusion_level}
include_insights: bool = True
include_notes: bool = True
max_tokens: Optional[int] = None
priority_weights: Optional[Dict[str, int]] = None # {type: weight}
def __post_init__(self):
"""Initialize default values."""
if self.sources is None:
@ -60,7 +61,7 @@ class ContextBuilder:
Generic ContextBuilder that can handle any parameters and build context
from sources, notebooks, insights, and notes.
"""
def __init__(self, **kwargs):
"""
Initialize ContextBuilder with flexible parameters.
@ -78,20 +79,20 @@ class ContextBuilder:
self.params = kwargs
# Extract commonly used parameters
self.source_id: Optional[str] = kwargs.get('source_id')
self.notebook_id: Optional[str] = kwargs.get('notebook_id')
self.include_insights: bool = kwargs.get('include_insights', True)
self.include_notes: bool = kwargs.get('include_notes', True)
self.max_tokens: Optional[int] = kwargs.get('max_tokens')
self.source_id: Optional[str] = kwargs.get("source_id")
self.notebook_id: Optional[str] = kwargs.get("notebook_id")
self.include_insights: bool = kwargs.get("include_insights", True)
self.include_notes: bool = kwargs.get("include_notes", True)
self.max_tokens: Optional[int] = kwargs.get("max_tokens")
# Context configuration
context_config_arg: Optional[ContextConfig] = kwargs.get('context_config')
context_config_arg: Optional[ContextConfig] = kwargs.get("context_config")
self.context_config: ContextConfig
if context_config_arg is None:
self.context_config = ContextConfig(
include_insights=self.include_insights,
include_notes=self.include_notes,
max_tokens=self.max_tokens
max_tokens=self.max_tokens,
)
else:
self.context_config = context_config_arg
@ -100,73 +101,72 @@ class ContextBuilder:
self.items: List[ContextItem] = []
logger.debug(f"ContextBuilder initialized with params: {list(kwargs.keys())}")
async def build(self) -> Dict[str, Any]:
"""
Build context based on provided parameters.
Returns:
Dict containing the built context with metadata
"""
try:
logger.info("Starting context building")
# Clear existing items
self.items = []
# Build context based on parameters
if self.source_id:
await self._add_source_context(self.source_id)
if self.notebook_id:
await self._add_notebook_context(self.notebook_id)
# Process any additional custom parameters
await self._process_custom_params()
# Apply post-processing
self.remove_duplicates()
self.prioritize()
if self.max_tokens:
self.truncate_to_fit(self.max_tokens)
# Format and return response
return self._format_response()
except Exception as e:
logger.error(f"Error building context: {str(e)}")
raise DatabaseOperationError(f"Failed to build context: {str(e)}")
async def _add_source_context(
self,
source_id: str,
inclusion_level: str = "insights"
self, source_id: str, inclusion_level: str = "insights"
) -> None:
"""
Add source and its insights to context.
Args:
source_id: ID of the source
inclusion_level: "insights", "full content", or "not in"
"""
if inclusion_level == "not in":
return
try:
# Ensure source ID has table prefix
full_source_id = (
source_id if source_id.startswith("source:")
else f"source:{source_id}"
source_id if source_id.startswith("source:") else f"source:{source_id}"
)
source = await Source.get(full_source_id)
if not source:
logger.warning(f"Source {source_id} not found")
return
# Determine context size based on inclusion level
context_size: Literal["short", "long"] = "long" if "full content" in inclusion_level else "short"
context_size: Literal["short", "long"] = (
"long" if "full content" in inclusion_level else "short"
)
source_context = await source.get_context(context_size=context_size)
# Add source item
@ -175,15 +175,17 @@ class ContextBuilder:
id=source.id or "",
type="source",
content=source_context,
priority=priority
priority=priority,
)
self.add_item(item)
# Add insights if requested and available
if self.include_insights and "insights" in inclusion_level:
insights = await source.get_insights()
for insight in insights:
insight_priority = (self.context_config.priority_weights or {}).get("insight", 75)
insight_priority = (self.context_config.priority_weights or {}).get(
"insight", 75
)
insight_item = ContextItem(
id=insight.id or "",
type="insight",
@ -191,24 +193,24 @@ class ContextBuilder:
"id": insight.id,
"source_id": source.id,
"insight_type": insight.insight_type,
"content": insight.content
"content": insight.content,
},
priority=insight_priority
priority=insight_priority,
)
self.add_item(insight_item)
logger.debug(f"Added source context for {source_id}")
except NotFoundError:
logger.warning(f"Source {source_id} not found")
except Exception as e:
logger.error(f"Error adding source context for {source_id}: {str(e)}")
raise
async def _add_notebook_context(self, notebook_id: str) -> None:
"""
Add notebook content based on context configuration.
Args:
notebook_id: ID of the notebook
"""
@ -216,7 +218,7 @@ class ContextBuilder:
notebook = await Notebook.get(notebook_id)
if not notebook:
raise NotFoundError(f"Notebook {notebook_id} not found")
# Process sources from context config or get all
config_sources = self.context_config.sources
if config_sources:
@ -242,134 +244,130 @@ class ContextBuilder:
for note in notes:
if note.id:
await self._add_note_context(note.id, "full content")
logger.debug(f"Added notebook context for {notebook_id}")
except Exception as e:
logger.error(f"Error adding notebook context for {notebook_id}: {str(e)}")
raise
async def _add_note_context(
self,
note_id: str,
inclusion_level: str = "full content"
self, note_id: str, inclusion_level: str = "full content"
) -> None:
"""
Add note to context.
Args:
note_id: ID of the note
inclusion_level: "full content" or "not in"
"""
if inclusion_level == "not in":
return
try:
# Ensure note ID has table prefix
full_note_id = (
note_id if note_id.startswith("note:")
else f"note:{note_id}"
)
full_note_id = note_id if note_id.startswith("note:") else f"note:{note_id}"
note = await Note.get(full_note_id)
if not note:
logger.warning(f"Note {note_id} not found")
return
# Get note context
context_size: Literal["short", "long"] = "long" if "full content" in inclusion_level else "short"
context_size: Literal["short", "long"] = (
"long" if "full content" in inclusion_level else "short"
)
note_context = note.get_context(context_size=context_size)
# Add note item
priority = (self.context_config.priority_weights or {}).get("note", 50)
item = ContextItem(
id=note.id or "",
type="note",
content=note_context,
priority=priority
id=note.id or "", type="note", content=note_context, priority=priority
)
self.add_item(item)
logger.debug(f"Added note context for {note_id}")
except NotFoundError:
logger.warning(f"Note {note_id} not found")
except Exception as e:
logger.error(f"Error adding note context for {note_id}: {str(e)}")
async def _process_custom_params(self) -> None:
"""Process any additional custom parameters."""
# Hook for future extensions - can be overridden in subclasses
# or used to process additional kwargs
for key, value in self.params.items():
if key.startswith('custom_'):
if key.startswith("custom_"):
logger.debug(f"Processing custom parameter: {key}={value}")
# Custom processing logic can be added here
def add_item(self, item: ContextItem) -> None:
"""
Add a ContextItem to the builder.
Args:
item: ContextItem to add
"""
self.items.append(item)
logger.debug(f"Added item {item.id} with priority {item.priority}")
def prioritize(self) -> None:
"""Sort items by priority (higher priority first)."""
self.items.sort(key=lambda x: x.priority, reverse=True)
logger.debug(f"Prioritized {len(self.items)} items")
def truncate_to_fit(self, max_tokens: int) -> None:
"""
Remove items if total token count exceeds limit.
Args:
max_tokens: Maximum allowed tokens
"""
if not max_tokens:
return
total_tokens = sum(item.token_count or 0 for item in self.items)
if total_tokens <= max_tokens:
logger.debug(f"Token count {total_tokens} within limit {max_tokens}")
return
logger.info(f"Truncating from {total_tokens} to {max_tokens} tokens")
# Remove items from the end (lowest priority) until under limit
current_tokens = total_tokens
removed_count = 0
while current_tokens > max_tokens and self.items:
removed_item = self.items.pop()
current_tokens -= (removed_item.token_count or 0)
current_tokens -= removed_item.token_count or 0
removed_count += 1
logger.info(f"Removed {removed_count} items, final token count: {current_tokens}")
logger.info(
f"Removed {removed_count} items, final token count: {current_tokens}"
)
def remove_duplicates(self) -> None:
"""Remove duplicate items based on ID."""
seen_ids = set()
deduplicated_items = []
for item in self.items:
if item.id not in seen_ids:
deduplicated_items.append(item)
seen_ids.add(item.id)
removed_count = len(self.items) - len(deduplicated_items)
self.items = deduplicated_items
if removed_count > 0:
logger.debug(f"Removed {removed_count} duplicate items")
def _format_response(self) -> Dict[str, Any]:
"""
Format the final response.
Returns:
Formatted context response
"""
@ -377,7 +375,7 @@ class ContextBuilder:
sources = []
notes = []
insights = []
for item in self.items:
if item.type == "source":
sources.append(item.content)
@ -385,10 +383,10 @@ class ContextBuilder:
notes.append(item.content)
elif item.type == "insight":
insights.append(item.content)
# Calculate total tokens
total_tokens = sum(item.token_count or 0 for item in self.items)
response = {
"sources": sources,
"notes": notes,
@ -402,66 +400,63 @@ class ContextBuilder:
"config": {
"include_insights": self.include_insights,
"include_notes": self.include_notes,
"max_tokens": self.max_tokens
}
}
"max_tokens": self.max_tokens,
},
},
}
# Add notebook_id if provided
if self.notebook_id:
response["notebook_id"] = self.notebook_id
logger.info(f"Built context with {len(self.items)} items, {total_tokens} tokens")
logger.info(
f"Built context with {len(self.items)} items, {total_tokens} tokens"
)
return response
# Convenience functions for common use cases
async def build_notebook_context(
notebook_id: str,
context_config: Optional[ContextConfig] = None,
max_tokens: Optional[int] = None
max_tokens: Optional[int] = None,
) -> Dict[str, Any]:
"""
Build context for a notebook.
Args:
notebook_id: ID of the notebook
context_config: Optional context configuration
max_tokens: Optional token limit
Returns:
Built context
"""
builder = ContextBuilder(
notebook_id=notebook_id,
context_config=context_config,
max_tokens=max_tokens
notebook_id=notebook_id, context_config=context_config, max_tokens=max_tokens
)
return await builder.build()
async def build_source_context(
source_id: str,
include_insights: bool = True,
max_tokens: Optional[int] = None
source_id: str, include_insights: bool = True, max_tokens: Optional[int] = None
) -> Dict[str, Any]:
"""
Build context for a single source.
Args:
source_id: ID of the source
include_insights: Whether to include insights
max_tokens: Optional token limit
Returns:
Built context
"""
builder = ContextBuilder(
source_id=source_id,
include_insights=include_insights,
max_tokens=max_tokens
source_id=source_id, include_insights=include_insights, max_tokens=max_tokens
)
return await builder.build()
@ -470,33 +465,31 @@ async def build_mixed_context(
source_ids: Optional[List[str]] = None,
note_ids: Optional[List[str]] = None,
notebook_id: Optional[str] = None,
max_tokens: Optional[int] = None
max_tokens: Optional[int] = None,
) -> Dict[str, Any]:
"""
Build context from mixed sources.
Args:
source_ids: List of source IDs
note_ids: List of note IDs
notebook_id: Optional notebook ID
max_tokens: Optional token limit
Returns:
Built context
"""
context_config = ContextConfig(max_tokens=max_tokens)
# Configure sources
if source_ids:
context_config.sources = {sid: "insights" for sid in source_ids}
# Configure notes
# Configure notes
if note_ids:
context_config.notes = {nid: "full content" for nid in note_ids}
builder = ContextBuilder(
notebook_id=notebook_id,
context_config=context_config,
max_tokens=max_tokens
notebook_id=notebook_id, context_config=context_config, max_tokens=max_tokens
)
return await builder.build()
return await builder.build()