Drive create_workspace against a real database and read the column back,
proving the global switch survives the INSERT for both enabled and
disabled states.
Set knowledge_store_enabled = config.KNOWLEDGE_STORE_ENABLED on new
workspaces so a freshly created workspace joins the git-native path
immediately instead of being left behind on the legacy store.
Assert normalize_filename, safe_folder_segment, and allocate_path
disambiguation stay within 255 bytes for multibyte titles, and that a
derived name writes where the raw title does not.
Truncate filenames and folder segments on a UTF-8 byte budget (cut on a
codepoint boundary, preserving the extension and disambiguation suffix)
so multibyte titles no longer emit paths that exceed ext4's per-component
limit and fail the seed with ENAMETOOLONG.
The end-of-turn subject generator reuses the agent's streaming llm, so its
tokens streamed into the user's reply and landed as a trailing "docs: ..."
line. Tag the call surfsense:internal, which the chat-model stream handler
already drops, matching the memory-rewrite/automation-draft internal calls.
Locks the manually-verified writer scenarios into regression coverage:
- A re-sync whose fresh metadata dropped the PATH_MARKER overwrites in place
via the durable `path` column instead of authoring a second path and forking
a duplicate (the original bug); plus the marker-present overwrite case.
- Removing a folder prunes its directory off disk, not just the tree rows, so a
deleted folder no longer lingers as a hollow shell.
A re-sync rewrites a document's metadata with fresh connector fields that
carry no PATH_MARKER, so ingest lost the file's path and authored a new one,
forking a duplicate on every re-index. Preserve the marker when updating an
existing row (_carry_store_marker) and, as a durable fallback, resolve the
recorded path from the `path` column when the marker is absent
(_recorded_virtual_path).
Also:
- Defer chunking to the store's indexer on flipped workspaces
(index_unless_store_owns) across the upload, note, obsidian, google-drive,
and local-folder writers, so a recorded batch is not double-chunked.
- Prune directories a removal leaves empty, so a deleted folder no longer
lingers on disk as a hollow shell.
- Treat .keep folder markers as structure not content in migration parity,
and skip re-embedding already-converged rows on a full index_tree rebuild.
An agent's empty folder lands in git as a bare .keep marker. The indexer
skipped it as a blank blob and only reconciled folders on a full rebuild, so
the folder had no row until then; the commit-time projection never reconciled
folders at all. Both paths now run a shared reconcile_tree_folders against the
tree snapshot, so an empty folder gets its row instantly at commit and the
incremental and full paths cannot drift.
GitTreeBackend.amkdir/armdir carry the .keep in and out so an agent's empty
folder enters the turn's diff the same way the facade's create_folder does.
Now that a note keeps its Postgres-owned title across a move, _resolve_by_title
could match a relocated row by its stale title and silently merge a fresh note
into it. Restrict that legacy fallback to rows with no path and no marker — the
unlocated rows it exists for. Move tests key by path, the field unique across a
move.
Assert the update tool reads the document then POSTs the body to the editor
save endpoint with the existing title pinned, so the repointed path is guarded.
Wire the remaining direct callers through the facade so a git-backed workspace
sees them: note create, circleback and extension ingests record the prepared
document; an obsidian rename records the move after the row commits.
Git owns the body, Postgres owns metadata, on-disk front-matter deferred — the
settled part, already the running behavior (C1/C5). The clean switch and the
record_*->intent rename are the Phase 5 cut's re-org, not a now-task: the cut
deletes the legacy arm, and the dual-run behind flags is correct meanwhile.
The MCP surfsense_update_document tool was the only live caller of
PUT /documents/{id}; it wrote the deprecated content column with no reindex,
and 409'd on git-backed workspaces. Repoint it at the wired editor save path
(source_markdown -> facade -> git + projection + reindex), which works on
flagged and unflagged workspaces alike, then delete the endpoint, its unused
DocumentUpdate schema, and its orphaned frontend (updateDocument service and
updateDocumentMutationAtom). One edit path, one source of truth.
Every path helper the agent runtime exposed already lives in
app.knowledge_store.paths, so path_resolver was a second, drifting copy of
the store's path law. Point all 28 importers at the package and delete the
425-line duplicate.
This also closes a layering inversion: index/{converge,rows,project} were
importing up into the agent runtime for path helpers that belong to their
own module.
The one behavioural difference was parse_documents_path, which strips the
.xml extension and " (<id>)" suffix to form a title — the contract the
indexer and its tests depend on. It moves to paths/legacy.py (it is xml
aware, so legacy) and becomes the exported one; store_path's unused raw
splitter is dropped.
virtual_path_to_doc is now the healed, column-first resolver everywhere,
a strict superset of the old hash/suffix/title lookup.
Folder create, rename, move and delete now record to git after the row
op, through thin module verbs (record_created_folder, record_moved_folder,
record_removed_folder, folder_virtual_path). Routes never spell a path or
bind a workspace; every verb self-guards, so an unflipped workspace is
untouched.
Rename and move capture the old path before mutating the row, then record
the move: the row is already at its new name, so the in-place reparent
no-ops and git still follows, id kept.
Delete drops only the folder's .keep markers (remove_folder_markers), never
its files. The incremental indexer prunes a row the moment its file leaves
the tree, so removing documents here would race the purge task that owns
their chunks and blobs. The markers are the resurrection gap; the purge
owns the documents.
Two blockers on routing folder CRUD through the facade are cleared.
move_folder renames the folder row in place (reparent_folder) before
reconcile, so a rename or reparent keeps the folder id instead of
prune-then-create churning it; child rows follow on parent_id, and
reconcile then finds the row already at the live chain and leaves it be.
The seed materializes each empty leaf folder as a .keep, since git holds
no empty directory. Only the leaf of an empty chain carries the marker;
its ancestors ride along on that path, and a folder a document already
lives in needs none.
The three facade writers that choose a name — save_document,
ingest_documents, move_documents — now derive through the naming law
(allocate_path / normalize_filename, "(2)" collisions) rather than the
legacy .xml derivation. A flipped workspace stops creating the very .xml
debt the seed heals; without this, every new note or rename re-stamped
.xml, and healing only ever touches files that already exist.
Occupancy comes from the git tree, the one authority on which files
exist, and a row's own file is excluded so a re-derivation after a lost
marker cannot collide the document with itself. The three sites switch
together: a partial swap forks a document between .md and .xml and breaks
the did-not-move check. doc_to_virtual_path / virtual_path_of stay for
the resolver and unflipped kb_postgres; only the flipped facade authors
.md.
The document-move handlers now record the move on the store after the
folder_id change, so a flipped workspace's move reaches git and a rebuild
finds the file at its new folder instead of resurrecting the old path. A
no-op on an unflipped workspace, since the verb self-guards per workspace;
bulk move groups by workspace because a move to root can span several and
each store binds to one.
_relocation_of reparents through the same derivation as the rest of the
live write path, so a move never forks the spelling. Folder CRUD stays on
folder_service for now: routing it onto the facade verbs waits on the seed
materializing .keep for existing empty folders (whole-workspace reconcile
would prune them) and on an id-preserving folder rename.
An import-boundary test pins that nothing under app/ outside the module
reaches the transaction, the engines, or the path submodules, so a second
writer or a second path spelling cannot creep back past the facade. The
package root stops re-exporting Transaction, which was the one way to reach
it without tripping the guard.
A symmetry test asserts the load-bearing invariant: a path authored into
git projects to a row whose path is the same string git stores, and that
path resolves back to the one row that authored it.
Folders now start in the store facade, empty ones included. StorePath
reserves .keep so a folder marker can never be authored as a document,
and three facade verbs each record one revision: create_folder writes
path/.keep, remove_folder removes the whole subtree, move_folder moves
every descendant so document ids survive the rename.
index/folders.py derives the folders rows from the union of document
parents and .keep folders and prunes the rows no path needs. It runs on
every folder verb (immediate) and on the full rebuild, gated on a clean
run so a mid-rollback session is never touched. A pruned row replicates
to the UI via Zero, closing the Phase-6 gap where an emptied folder
lingered.
migrate_workspace stops deriving .xml names. A row that already records
an authored-once path keeps it; an unmarked row is authored a fresh .md
path via allocate_path, in created_at then id order so same-title
collisions number the same way on every re-seed. Recorded paths are
reserved before the unmarked rows are authored, so a fresh name never
lands on one a marked row owns. This file now reads its path vocabulary
from app.knowledge_store.paths, off the legacy shim.
Promote the virtual path off document_metadata onto documents.path, the
authored-once identity the path law resolves on. Migration 177 is an
instant ADD COLUMN plus a non-unique partial index on the non-NULL rows
-- no backfill, since the mandatory rewrite is what killed the ~21-day
chunk migration. Rows heal on every write (projection upsert, service
save/move, the seeder); resolution reads the column first and falls back
to the marker for rows written before it existed. The unique partial
index is deferred to a runbook step run once the fleet is healed.
Four concerns become four files behind the same public surface: layout
(on-disk), store_path (the validated /documents value object), naming
(sanitize + allocate), resolve (path -> row), and legacy (the .xml
title derivation). The legacy renderers are now one file to delete at
the cut instead of lines threaded through the rest. No behavior change.
DocumentHybridSearchRetriever called embedding_model.embed() directly from its
coroutines. A local sentence-transformers model runs a synchronous forward pass,
so that call stalls the event loop and every other request sharing the worker
for its duration.
Every sibling already offloads it: ChucksHybridSearchRetriever does in both
vector_search and hybrid_search, and so do ConnectorService._combined_rrf and
the multi-agent chat retrieval path. This retriever was the one that did not.
Scope note: today these two lines are not reachable. vector_search and
full_text_search have no callers, and hybrid_search's only caller always passes
a precomputed query_embedding. So this is a consistency fix that keeps the next
caller from silently reintroducing the stall -- not a measured speedup, and the
PR should not be read as one.
The podcast package already centralises this: invoke_json strips fences,
validates against a Pydantic model, retries on a brace-scanned substring and
raises a typed StructuredOutputError, logging only a bounded snippet of the
reply. Its docstring says it exists so every generation node validates replies
the same way, but nothing outside app/podcasts ever called it.
The video presentation agent had grown its own copy of the same algorithm.
That copy reported failures with print, and one of those prints wrote the
entire model reply to stdout -- the reply being a summary of whatever document
the user fed in, so a parse failure leaked document content into worker logs.
Move the helper to app/utils/structured_output.py, where a general-purpose LLM
parsing utility belongs rather than under a feature package, and call it from
the video agent. Theme assignment keeps its degrade-to-round-robin behaviour
but now logs with a traceback instead of a bare print, because that branch also
swallows transport errors and a print made those indistinguishable from a
malformed reply.
The video presentation agent built its Kokoro pipeline with a literal
lang_code="a" and took its voice from a hand-rolled provider map that only
ever returned American English. Slides generated from Japanese or Chinese
source content were therefore narrated by an English voice reading text it
could not pronounce.
The podcast pipeline already solved this: a voice catalog with per-language
rosters, a BCP-47 tag normaliser, and a TTS port whose Kokoro adapter maps the
tag to the right pipeline and caches one pipeline per language. The video path
simply sat outside that infrastructure. Route it through the same port instead
of duplicating a second, English-only implementation.
The language now comes from the model that writes the slides, which is the
only component that knows what language it wrote in. An absent or unusable tag
falls back to a new VIDEO_PRESENTATION_DEFAULT_LANGUAGE and finally to English,
so a bad value degrades instead of failing the render.
app/services/kokoro_tts_service.py is removed: routing through the port left it
without a caller, and it also pulled kokoro and torch into the agent's import
chain at module scope.
