mirror of
https://github.com/lfnovo/open-notebook.git
synced 2026-05-01 21:00:43 +00:00
feat: content-type aware chunking and unified embedding (#444)
* feat: content-type aware chunking and unified embedding - Add chunking.py with HTML, Markdown, and plain text detection - Add embedding.py with mean pooling for large content - Create dedicated commands: embed_note, embed_insight, embed_source - Use fire-and-forget pattern for embedding via submit_command() - Refactor rebuild_embeddings_command to delegate to individual commands - Remove legacy commands and needs_embedding() methods - Reduce chunk size to 1500 chars for Ollama compatibility - Update CLAUDE.md documentation for new architecture Fixes #350, #142 * fix: address code review issues - Note.save() now returns command_id for tracking embedding jobs - Add length check after generate_embeddings() to fail fast on mismatch - Add numpy as explicit dependency (was transitive) - Remove hardcoded chunk sizes from docstrings * docs: address code review comments - Rename "SYNC PATH" to "DOMAIN MODEL PATH" in embedding router - Add test_chunking.py and test_embedding.py to Testing Strategy - Clarify auto-embedding behavior for each domain model * fix: clean thinking tags from prompt graph output Adds clean_thinking_content() to prompt.py to handle extended thinking models that return <think>...</think> tags. This fixes empty titles when saving notes from chat. * chore: remove local docker-compose from git * fix(frontend): handle null parent_id in search results Add defensive check for null parent_id in search results to prevent "Cannot read properties of null (reading 'split')" error. This can happen with orphaned records in the database. * fix: cascade delete embeddings and insights when source is deleted When deleting a Source, now also deletes associated: - source_embedding records - source_insight records This prevents orphaned records that cause null parent_id errors in vector search results. * fix: add cleanup for orphan embedding/insight records in migration 10 Deletes source_embedding and source_insight records where the linked source no longer exists (source.id = NONE). * chore: bump esperanto to 2.16 Increases ctx_num for Ollama models to accommodate larger notebook context windows. See: https://github.com/lfnovo/esperanto/pull/69
This commit is contained in:
parent
78b9024ad3
commit
d8006ff5cb
26 changed files with 1915 additions and 680 deletions
|
|
@ -33,7 +33,7 @@ Two base classes support different persistence patterns: **ObjectModel** (mutabl
|
|||
- `add_insight()`: Generate and store insights with embeddings
|
||||
|
||||
- **Note**: Standalone or linked notes
|
||||
- `needs_embedding()`: Always True (searchable)
|
||||
- `save()`: Submits `embed_note` command after save (fire-and-forget)
|
||||
- `add_to_notebook()`: Link to notebook
|
||||
|
||||
- **SourceInsight, SourceEmbedding**: Derived content models
|
||||
|
|
@ -55,7 +55,7 @@ Two base classes support different persistence patterns: **ObjectModel** (mutabl
|
|||
|
||||
- **Async/await**: All DB operations async; always use await
|
||||
- **Polymorphic get()**: `ObjectModel.get(id)` determines subclass from ID prefix (table:id format)
|
||||
- **Auto-embedding**: `save()` generates embeddings if `needs_embedding()` returns True
|
||||
- **Fire-and-forget embedding**: Models submit embed_* commands after save via `submit_command()` (non-blocking)
|
||||
- **Nullable fields**: Declare via `nullable_fields` ClassVar to allow None in database
|
||||
- **Timestamps**: `created` and `updated` auto-managed as ISO strings
|
||||
- **Fire-and-forget jobs**: `source.vectorize()` returns command_id without waiting
|
||||
|
|
@ -75,14 +75,17 @@ Two base classes support different persistence patterns: **ObjectModel** (mutabl
|
|||
- **RecordModel singleton**: __new__ returns existing instance; call `clear_instance()` in tests
|
||||
- **Source.command field**: Stored as RecordID; auto-parsed from strings via field_validator
|
||||
- **Text truncation**: `Note.get_context(short)` hardcodes 100-char limit
|
||||
- **Embedding async**: Only Note and SourceInsight embed on save; Source too large (uses async job)
|
||||
- **Auto-embedding behavior**:
|
||||
- `Note.save()` → auto-submits `embed_note` command
|
||||
- `Source.save()` → does NOT auto-submit (must call `vectorize()` explicitly)
|
||||
- `Source.add_insight()` → auto-submits `embed_insight` command
|
||||
- **Relationship strings**: Must match SurrealDB schema (reference, artifact, refers_to)
|
||||
|
||||
## How to Add New Model
|
||||
|
||||
1. Inherit from ObjectModel with table_name ClassVar
|
||||
2. Define Pydantic fields with validators
|
||||
3. Override `needs_embedding()` if searchable
|
||||
3. Override `save()` to submit embedding command if searchable (use `submit_command("embed_*", id)`)
|
||||
4. Add custom methods for domain logic (get_X, add_to_Y)
|
||||
5. Implement `_prepare_save_data()` if custom serialization needed
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue