index_revision promised an argument it does not take — its docstring had
to open by disclaiming it — and reindex is the same verb the legacy
per-document editor path already uses, which is the writer this phase
guards against.
Fossil's rebuild was the obvious alternative and is wrong here: the run
upserts and prunes, and document ids survive it because they reach the
browser through the Zero publication. Nothing is wiped but chunks.
The two differ only in scope, so the names say scope: index_changes
folds in what moved since the stamp, index_tree reconciles against the
whole tree and is the only one that can notice a deletion it never saw.
Celery task names are untouched; they are the wire contract.
Also corrects the umbrella plan, which described this as "wipe +
rebuild" while 04-derived-index.md said document rows are never wiped.
The core package was clean apart from two files: everything under
knowledge_store imports only itself and app.config, while indexer.py
reached into app.db, the indexing pipeline and the agents middleware.
Per ADR 0002 that file is a driven consumer, not core, and nothing in
the layout said so.
Moves it to knowledge_store/index/ as converge.py + queue.py, and the
three celery entry points to celery_tasks/knowledge_store/. Task names,
queue routes and beat entries are unchanged; only module paths move.
index/__init__.py deliberately re-exports nothing. queue.py is a
writer's last step and has to stay cheap to import, so a convenience
re-export would put the ORM and the pipeline on every save's import
path. A test pins that, and fails if the re-export comes back.
This check is what replaces reading migration reports by hand once
workspaces are flipped, so an "ok" it reports has to mean git and
Postgres actually agree. Runs against real repos and rows: a seeded
workspace reads ok, an unseeded document reads drift, an unflipped
workspace is not checked, and one failing workspace still leaves the
next one checked.
Mutation-verified: collapsing _status to a constant "ok" fails.
enqueue_index deliberately swallows broker failures so a committed save
never fails on a queue problem. Untested, that swallow makes a typo
look exactly like a broker outage: nothing enqueues and nothing
complains. Covers the dispatch, the kill switch, the non-numeric id
drop, and the swallowed connection error.
The sweep is the only recovery for indexing lost to a broker or worker,
and it was the one piece of phase 4 with no test at all. Pins the
flipped-only filter, the skip when a stamp is level or a store is
missing, the enqueue cap, and the routing split where a never-indexed
workspace goes to the rebuild task rather than the fast queue.
Both of those last two were verified by mutation: dropping the flipped
filter and inverting the routing each fail a test.
The umbrella's subplan index still said PLANNED for phases 2-5 while
the inline sections said SHIPPED, and phase 5 had no status line at
all. Both now record: 1-4 shipped, 5 tooling shipped with fleet flips
and cut-time deletion pending, C2 read_file citation envelope the one
open phase-2 item.
Lands the Phase 4 derived-index work and aligns it with the shipped
conventions: repo paths keep the documents/ root, flag checks go through
the per-workspace knowledge_store_enabled_for, the retitle mechanism is
folded into the batched revision recorder, lock expiry stays loud, and
the migration is renumbered 175 -> 176 on top of the flip-flag one.
Incremental index tasks now retry on lock contention so saves landing
mid-rebuild don't wait for the hourly sweep.
The editor reindex titles a document from its first heading while the indexer titles it from the filename, so both running flips the title on every save; guarded inside _reindex_document so neither call site can miss it. Version restore returns 409, since it rewrites content without a revision and the next reindex would revert it. Unrelated: all three version routes passed a User where check_permission expects an AuthContext, so every one of them 500'd.
Enqueue-only and never raising: the content is already committed, and the drift sweep is the backstop. Also drops the stale blob when a retitle moves a document, which otherwise turned one document into two files the moment an indexer existed.
Hourly with a per-run enqueue cap, not daily: KNOWLEDGE_STORE_ENABLED is process-global, so the first sweep after enabling it is a fleet-wide backfill. Rebuilds route to the connectors queue; the per-save task stays on the fast queue.
index_revision and reindex share one convergence body, so the incremental and rebuild paths cannot drift. Rows are adopted by path and pruned only when they carry the ownership marker, so an upload is not duplicated and a connector row is not deleted. A partial failure withholds the stamp; an intentional skip still stamps.
Spans are located by ordered cursor search at read time rather than stored in the cached value, so no chunker_version bump and no fleet-wide re-embed. Reused chunks now carry the new span, because unchanged text still moves when a paragraph is inserted above it.
One migration for both tables so the two columns cannot become two Alembic heads. No backfill: NULL last_indexed_revision is what makes the drift sweep pick a workspace up, and NULL spans repopulate on the next index.
--yes --flip turns a workspace git-native strictly after its report is
ok; --unflip --workspace N is the per-workspace rollback. Verified end
to end on the dev database: seed, flip, column true, unflip. Plan work
item 4 marked shipped.
The factory resolves knowledge_store_enabled_for once and the whole turn
inherits it (backend resolver, persistence middleware, compiled-graph
cache key - the flag must rotate cached graphs or a flipped workspace
would keep serving its pre-flip graph). The recorder and the disconnect
safety-net check per call; the janitor stays global, pruning is
workspace-agnostic.
workspaces.knowledge_store_enabled (migration 175, default false) makes
the git-native rollout progressive: flip one workspace, watch it live,
continue. knowledge_store_enabled_for(workspace_id) owns the verdict -
global master switch read live (disaster lever acts immediately) AND the
workspace row, cached 30s per process.
Loops workspaces through migrate_workspace with a fresh session each so
one failure cannot poison the pass; every report lands in an append-only
JSONL audit file. Dry run by default (--yes to seed), exits non-zero if
any workspace is not ok. Verified against the dev database: the piloted
workspace reports parity ok.
The Phase-3 recorder and Phase-5 seeder both record full virtual paths
(documents/...); the local pilot seed confirmed both sides agree. Amend
C1 so the contract describes what actually ships.
migrate_workspace maps documents through the live path rules and records
one migration-authored seed revision; re-seeding converges (orphans of
Postgres deletions are removed) so seed -> catch-up -> flip is safe.
Parity compares content addresses, no file reads. Neither entry point
raises: any failure lands in MigrationReport.error, so one broken
workspace cannot abort a fleet run.
Release suppressed every LockError, so a hold that outlived the TTL -
exclusivity silently lost mid-write - left no trace. Expiry now raises
KnowledgeStoreLockError after a clean write; an error from the write
itself still wins and is never masked.
porcelain.add rewrites the whole index per call, so staging an n-file
revision cost n index rewrites - a 2000-doc seed took 53s and would
outlive the 30s write-lock TTL. One batched add/remove per revision
makes recording linear: the same seed now lands in 0.6s.
Phase 5 redesigned after the 21-day-backfill lesson: seed commit +
byte parity replaces the reindex gate, and the seed revision is
adopted by the indexer, never incrementally indexed (new C7).
C2 amended: start_line/end_line stored at cut time for new chunks
(nullable, instant ALTER, no migration backfill); legacy chunks cite
at document level, enforced by a fail-closed normalizer strip; a
deadline-free daily fill job converges them most-used-first, matching
against the git blob all-or-nothing so no path yields wrong numbers.
C5 corrected: the embedding cache exists (indexing_pipeline/cache,
2026-06-12) and is the content-id reuse layer - extend it, never warm
it from legacy rows.
Every indexer converges there, so recording happens the moment markdown
becomes durable - one revision per sync batch, never blocked by later
chunking/embedding failures. Supersedes the per-upload recorder call.
record_markdown_files records many paths as one transaction; a save is
now a batch of one, and record_prepared_documents maps a sync batch to
a single revision. Unchanged content stays a natural no-op.
