mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-08-04 13:20:21 +00:00
Merge pull request #1650 from CREDO23/revert-kb-git-mvp
Revert "Merge pull request #1649 from CREDO23/kb_git_mvp"
This commit is contained in:
Thierry CH.
GitHub
commit
4e9225b1cd
119 changed files with 4654 additions and 13758 deletions
|
|
@ -1,170 +0,0 @@
|
|||
# ADR 0001: Git-native Knowledge Base (Git as source of truth, Postgres as derived index)
|
||||
|
||||
- **Status:** Proposed (brainstorm outcome — for team review)
|
||||
- **Date:** 2026-07-24
|
||||
- **Origin:** Rohan Verma's meeting proposal to pivot from the custom-built KB "file system" to a Git-based system due to persistent maintenance issues; Thierry Bakera to investigate.
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
### What we have today
|
||||
|
||||
SurfSense does **not** actually have a file system. It has a **virtual filesystem façade mapped onto Postgres rows**, used by the chat agent. The moving parts:
|
||||
|
||||
- Virtual `/documents/` namespace computed from DB rows — `surfsense_backend/app/agents/chat/runtime/path_resolver.py`
|
||||
- Read-side backend faking `ls`/`read`/`glob`/`grep` over Postgres — `.../filesystem/backends/kb_postgres.py`
|
||||
- Write-side "commit at end of turn" layer — `.../main_agent/middleware/kb_persistence/middleware.py`
|
||||
- **Three separate hand-rolled versioning/audit systems:**
|
||||
- `DocumentVersion` (user history) — `app/utils/document_versioning.py`
|
||||
- `DocumentRevision` / `FolderRevision` (agent revert snapshots) — `app/services/revert_service.py`
|
||||
- `AgentActionLog` (tool-call audit)
|
||||
- Supporting machinery: fractional indexing for ordering, move tracking, `content_hash` change detection, chunk reconciliation.
|
||||
|
||||
### The problem
|
||||
|
||||
The team has been hand-implementing — on top of a relational DB never designed for it — the exact primitives Git provides natively. That re-implementation is the source of the "persistent maintenance issues."
|
||||
|
||||
| Hand-rolled today | Git provides natively |
|
||||
|---|---|
|
||||
| `path_resolver` + folder tree | tree objects |
|
||||
| end-of-turn staged commit | atomic commits |
|
||||
| `DocumentVersion` snapshots | commit history |
|
||||
| `DocumentRevision` + `revert_service` | `git revert` / `reset` |
|
||||
| `AgentActionLog` | commit log / `blame` |
|
||||
| `content_hash` dedup | content-addressed blobs (SHA) |
|
||||
| fractional indexing / move tracking | tree diff / rename detection |
|
||||
|
||||
### Search stack (unchanged by this ADR)
|
||||
|
||||
Hybrid chunk search: pgvector (HNSW) + Postgres FTS + RRF, optional reranking. Chunking via Chonkie; incremental via `chunk_reconciler.py`. See `.../shared/retrieval/hybrid_search.py`.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
**Adopt Git as the single source of truth for all indexed KB content. Postgres becomes a derived, rebuildable index holding only chunks + embeddings.**
|
||||
|
||||
### Core model
|
||||
|
||||
```
|
||||
agent notes ─┐
|
||||
editor saves ─┤
|
||||
uploads ──────┤→ Git commit (source of truth) → indexer → Postgres (chunks + embeddings)
|
||||
Notion ───────┤
|
||||
Drive ────────┘
|
||||
(indexable connectors only)
|
||||
|
||||
Slack / Gmail (live connectors) ──→ queried at chat time, bypass storage entirely
|
||||
```
|
||||
|
||||
- **Git = truth** for everything that gets stored/indexed (agent/editor notes, uploads, and **indexable** connectors like Notion, Drive, Obsidian — the `is_indexable` connectors).
|
||||
- **Postgres = derived index only** (chunks + embeddings). It is a **cache**: it can be wiped and rebuilt from Git at any time via a single `reindex(workspace)` function.
|
||||
- **Live connectors (Slack, Gmail)** are never stored or indexed — they are queried live at chat time and are entirely out of scope for this design.
|
||||
- **Binary blobs** (original PDF/DOCX) stay in the existing local/Azure blob store (or Git-LFS later); Git holds the extracted markdown, not raw binaries.
|
||||
|
||||
### What changes for the agent
|
||||
|
||||
The agent's **tools are unchanged** (`ls`, `read`, `write`, `edit`, `mv`, `rm`). Only the backend behind them changes:
|
||||
|
||||
| Agent action | Backed by |
|
||||
|---|---|
|
||||
| File ops (`ls`/`read`/`write`/`edit`/`mv`/`rm`) | **Git** working tree (real files) |
|
||||
| Semantic search | **Postgres** (derived chunk/embedding index) |
|
||||
|
||||
- **Before:** `KBPostgresBackend` fakes files over Postgres rows.
|
||||
- **After:** a Git-working-tree backend operates on **real files**. `path_resolver` largely disappears (paths are real).
|
||||
- **Write flow:** agent edits → git working tree → **one commit at end of turn** (replaces the `kb_persistence` commit-to-Postgres step) → indexer refreshes Postgres chunks.
|
||||
|
||||
### What we delete
|
||||
|
||||
- The virtual-FS façade (`path_resolver`, `kb_postgres` staging).
|
||||
- `DocumentVersion`, `DocumentRevision` / `FolderRevision`, `revert_service` → replaced by git history + `git revert`.
|
||||
- Fractional indexing / bespoke move tracking → git tree operations.
|
||||
|
||||
This pivot is **mostly deletion**, which is the point.
|
||||
|
||||
---
|
||||
|
||||
## We are borrowing from authoritative sources, not inventing
|
||||
|
||||
Every decision traces to a proven, battle-tested reference. The only SurfSense-specific work is the *adaptation glue*.
|
||||
|
||||
| Decision | Borrowed from |
|
||||
|---|---|
|
||||
| Git = truth, Postgres = rebuildable cache | **Fossil SCM** — canonical artifacts + SQL tables as a pure cache, recomputed via `fossil rebuild` (production since 2007). https://fossil-scm.org/home/doc/trunk/www/theory1.wiki |
|
||||
| Content in git, metadata/index in a DB | **Gollum** (GitHub/GitLab wikis): "storage abstraction layer... only some data in the DB". https://github.com/gollum/gollum · https://docs.gitlab.com/17.5/development/wikis/ |
|
||||
| Silent commit-per-save, hide git from users | **kherad**. https://github.com/mohammadmaso/kherad |
|
||||
| Embeddings keyed by blob SHA, incremental (not rebuild) | **Coregit LLM Wiki** (https://coregit.dev/blog/llm-wiki-launch) + vector-index-as-cache best practices (LangChain RecordManager, LlamaIndex docstore). Cache key = `(model_version, content_hash)`. |
|
||||
| Python git engine | **dulwich** — pure Python, deploy-friendly, real wire protocol. https://github.com/jelmer/dulwich |
|
||||
| "Don't put a firehose in git" / git limits | "Git is not a database" critiques — degrades past ~500k–1M files, no query engine, no concurrency control. |
|
||||
| Future content model (`raw/`+`wiki/`, lint, contradiction-flagging) | **Karpathy's LLM Wiki**. https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f |
|
||||
| Future real-time collab | **Yjs** (Notion, Linear). https://github.com/yjs/yjs |
|
||||
| Future fact-level temporal memory | **Zep / Graphiti** (bi-temporal, "invalidate don't delete"). https://arxiv.org/html/2501.13956 |
|
||||
|
||||
### The counter-example we explicitly reject
|
||||
|
||||
**Wiki.js** git module = DB is truth, git is a two-way mirror. This creates two sources of truth and a reconciliation cursor; it has a known class of silent-sync bugs (see requarks/wiki discussion #7860). We use **one-way derivation** (git → Postgres), never two-way sync.
|
||||
|
||||
---
|
||||
|
||||
## Scope: v1 (keep it simple)
|
||||
|
||||
Ship the smallest thing that removes the maintenance pain:
|
||||
|
||||
1. **Git repo per workspace** (dulwich) holding indexed content as markdown.
|
||||
2. **One commit per agent turn / editor save.** No branches, no merge, no review workflow.
|
||||
3. **Delete** the three versioning systems; history/undo = git log + `git revert`.
|
||||
4. **pgvector stays**, rebuilt from git, keyed by blob SHA (point existing `chunk_reconciler` at blob SHA so unchanged files skip re-embedding).
|
||||
5. **Per-workspace lock/queue** around commits (git is single-writer; this is a data-integrity boundary, not a feature — non-negotiable).
|
||||
6. **One `reindex(workspace)` function** that wipes and rebuilds Postgres chunks from the git repo (the Fossil `rebuild` discipline — makes the system safe to ship early).
|
||||
|
||||
### Deferred (v2+, explicitly out of v1)
|
||||
|
||||
- Connect-your-own-remote (GitHub/GitLab) — free later because the repo is real git.
|
||||
- CRDT / Yjs real-time collaboration.
|
||||
- Review / merge workflows (kherad's reviewer layer).
|
||||
- Graphiti / bi-temporal fact graph.
|
||||
- Karpathy `raw/` + `wiki/` content model, contradiction-flagging, lint.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Large net **deletion** of bespoke code (the maintenance win).
|
||||
- Storage and search **decouple** → the team can improve semantic search independently (Rohan's stated goal).
|
||||
- Postgres becomes disposable/rebuildable → simpler recovery, fewer consistency bugs.
|
||||
- Unlocks future "user owns their KB as a real git repo" differentiator.
|
||||
|
||||
### Negative / risks
|
||||
|
||||
- **Concurrency:** git is single-writer per repo → requires the per-workspace lock (mitigated in v1).
|
||||
- **Repo hygiene:** many small commits → periodic `git gc`/repack (operational, manageable).
|
||||
- **Migration:** existing Postgres KBs must be exported into git repos once, preserving `unique_identifier_hash` mapping.
|
||||
- **Real-time UI (Zero):** currently driven by Postgres logical replication; still needs a git → Postgres projection to keep the web client live. This is *new* code that partially offsets deletions.
|
||||
|
||||
---
|
||||
|
||||
## Open questions (for team discussion)
|
||||
|
||||
1. **Zero / real-time UI:** confirm the git → Postgres projection path and whether Zero stays as-is.
|
||||
2. **Binaries:** keep blob store vs. adopt Git-LFS.
|
||||
3. **Migration cutover:** big-bang vs. per-workspace feature flag (recommend feature-flag rollout).
|
||||
4. **Merge UX later:** CRDT (Yjs) vs. review-gate (kherad) when multi-writer becomes a requirement.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: key file index (current implementation)
|
||||
|
||||
| Topic | Path |
|
||||
|---|---|
|
||||
| Virtual path resolver | `surfsense_backend/app/agents/chat/runtime/path_resolver.py` |
|
||||
| Virtual FS read backend | `.../filesystem/backends/kb_postgres.py` |
|
||||
| Virtual FS write commit | `.../main_agent/middleware/kb_persistence/middleware.py` |
|
||||
| Hybrid search | `.../shared/retrieval/hybrid_search.py` |
|
||||
| Chunk reconciliation | `surfsense_backend/app/indexing_pipeline/chunk_reconciler.py` |
|
||||
| User version history | `surfsense_backend/app/utils/document_versioning.py` |
|
||||
| Agent revert | `surfsense_backend/app/services/revert_service.py` |
|
||||
| ORM models | `surfsense_backend/app/db.py` |
|
||||
|
|
@ -1,64 +0,0 @@
|
|||
# ADR 0002: Knowledge core as Ports & Adapters (deepagents is an adapter, not the core)
|
||||
|
||||
- **Status:** Proposed (brainstorm outcome — for team review)
|
||||
- **Date:** 2026-07-28
|
||||
- **Relates to:** [ADR 0001](0001-git-native-knowledge-base.md) decides *what* the store is (git as source of truth, Postgres derived). This ADR decides the *shape* of the code around it.
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Should the git-backed store be built **as a deepagents backend**, or as a **standalone core** that deepagents consumes?
|
||||
|
||||
The same knowledge already has **multiple real consumers**:
|
||||
|
||||
- the chat agent (deepagents filesystem tools),
|
||||
- a **Knowledge Base REST API** (Rohan's mandate to standardize artifact generation and drop redundant UI dialogues),
|
||||
- the **vector-store sync** (the derived chunk/embedding index),
|
||||
- and, later, an **MCP server** and **connect-your-own-remote** (GitHub/GitLab).
|
||||
|
||||
If the core were a deepagents backend, every other consumer would have to route through an agent framework to touch the KB. That is the wrong dependency direction.
|
||||
|
||||
## Decision
|
||||
|
||||
**Structure the KB as Hexagonal / Ports & Adapters.** A framework-agnostic **knowledge core** exposes capabilities through **ports**; every consumer is an **adapter** at the edge; the core imports no consumer framework.
|
||||
|
||||
- **Core (inside the hexagon):** `KnowledgeStore` — versioned knowledge (content + history). Already framework-free (imports no deepagents).
|
||||
- **Driven port** (core → infrastructure): `VersionedContentEngine` → `GitContentEngine` (dulwich). Swappable, mirroring libgit2's pluggable backends.
|
||||
- **Driven consumer** (downstream of commits): the **vector-store sync / derived index** — subscribes to commits, one-way (git → Postgres). The core does not know it exists.
|
||||
- **Driving adapters** (world → core): the **deepagents backend** (build now), the **KB REST API** (later), **MCP / remote git** (future). The core is oblivious to all of them.
|
||||
- **Capabilities live once in the core; each adapter selects the subset it needs.** `ls`/`grep`/`glob` are offered for filesystem-shaped consumers (agent, MCP); document/version/diff verbs serve the REST API; commit-diff serves the sync. We never force one consumer's verbs on another, and never reimplement a capability per consumer.
|
||||
|
||||
## Borrowing, not inventing
|
||||
|
||||
| Element | Borrowed from |
|
||||
|---|---|
|
||||
| Framework-agnostic core, many adapters per port | **Cockburn — Ports & Adapters (Hexagonal)**: "there will typically be multiple adapters for any one port." https://alistair.cockburn.us/hexagonal-architecture |
|
||||
| Agnostic core + thin consumer layer | **Git's own plumbing/porcelain split** — low-level toolkit as building blocks, user commands on top. https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain |
|
||||
| Linkable core + pluggable storage backends for long-running services | **libgit2** (built precisely because forking the git binary is wrong for services) — same reasoning that chose **dulwich** behind `VersionedContentEngine`. |
|
||||
|
||||
## Scope — YAGNI (v1)
|
||||
|
||||
**Build now:** the **core** + the **deepagents adapter** + the **vector-store-sync driven consumer**. Nothing else.
|
||||
|
||||
- Design the ports so the **REST API** and **MCP/remote-git** adapters slot in later — but **do not build them now**.
|
||||
- **Grow the port surface on demand:** add a capability when an adapter needs it. No speculative methods.
|
||||
- The deepagents adapter should **reuse deepagents' own `FilesystemBackend`/git plumbing** for read-only structure ops (`ls`/`glob`/`grep`) rather than reimplementing them; writes + commit route through the core so the write lock, commit policy, and citation control have one home.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Consumers are decoupled: the KB REST API and MCP become **thin adapters**, not parallel rewrites of the KB.
|
||||
- The core is testable with no agent framework in the loop; deepagents itself becomes swappable.
|
||||
- Matches ADR 0001's "separate file management from the vector store/search" — the sync is just one driven consumer.
|
||||
|
||||
### Negative / cost
|
||||
|
||||
- One indirection (an adapter) over using `FilesystemBackend` directly — accepted because the consumers are real, not hypothetical (the YAGNI test for hexagonal).
|
||||
|
||||
## Open (deferred to their phases)
|
||||
|
||||
- **Citation model** for `read_file`: raw file vs chunk-rendered `[n]`. Rohan flagged current citations as poor — redesign candidate.
|
||||
- **Commit granularity**: per-turn (Aider-style) vs per-mutation.
|
||||
- **Turn isolation**: shared working tree + per-workspace lock vs per-turn worktree.
|
||||
|
|
@ -1,240 +0,0 @@
|
|||
# Git-native Knowledge Base — Umbrella Plan
|
||||
|
||||
> Master roadmap for pivoting the Knowledge Base from the custom virtual-filesystem-over-Postgres to **Git as the source of truth**. Each phase becomes its own subplan in this folder (`plans/git-native-kb/`).
|
||||
|
||||
This is the high-level roadmap. It is sequenced. Companion diagrams live in [`00b-diagrams.md`](00b-diagrams.md). The design rationale + references live in the ADR: [`docs/adr/0001-git-native-knowledge-base.md`](../../docs/adr/0001-git-native-knowledge-base.md).
|
||||
|
||||
> **SCOPE:** BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`), the real-time UI (Zero), and client apps (desktop, Obsidian, browser extension) are touched only where a phase forces it (Phase 6). A dedicated frontend/client umbrella comes LATER, once the backend is working.
|
||||
|
||||
> **Origin:** Rohan Verma's meeting proposal to pivot from the custom-built KB "file system" to a Git-based system due to persistent maintenance issues; Thierry Bakera to investigate. This umbrella is the outcome of that investigation.
|
||||
|
||||
## Positioning
|
||||
|
||||
The KB today has **no real filesystem** — it is a *virtual* `/documents/` namespace faked over Postgres rows, plus three hand-rolled versioning/audit systems. That re-implements — badly — what Git provides natively (tree, atomic commits, history, revert, content-addressed dedup). The pivot makes **Git the single source of truth for all indexed content** and demotes **Postgres to a derived, rebuildable search index (chunks + embeddings only)**. Net effect: large code **deletion**, storage and search **decoupled** (so search can improve independently — Rohan's stated goal), and a real git repo per workspace (unlocking "bring your own remote" later).
|
||||
|
||||
## Target architecture (git = truth, Postgres = derived index)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph WRITE["Write path (one path for everything indexed)"]
|
||||
AG["Agent notes"] --> GIT
|
||||
ED["Editor saves (Plate.js)"] --> GIT
|
||||
UP["Uploads (extracted markdown)"] --> GIT
|
||||
NOT["Indexable connectors: Notion / Drive / Obsidian"] --> GIT
|
||||
end
|
||||
GIT["Git repo per workspace (SOURCE OF TRUTH)\ncommit per turn/save · dulwich · per-workspace lock"]
|
||||
GIT --> IDX["Indexer: diff tree → changed blobs\n(embed keyed by blob SHA)"]
|
||||
IDX --> PG[("Postgres = DERIVED index\nchunks + embeddings only (rebuildable)")]
|
||||
subgraph READ["Agent"]
|
||||
FS["file ops: ls/read/write/edit/mv/rm"] --> GIT
|
||||
SR["semantic search"] --> PG
|
||||
end
|
||||
LIVE["Live connectors: Slack / Gmail"] -.->|queried at chat time, never stored| SKIP["(bypass storage entirely)"]
|
||||
BLOB[("Blob store / Azure — original binaries (unchanged)")]
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Current (to-be-replaced) architecture — virtual FS over Postgres</summary>
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
AG["Agent tools ls/read/write/edit/mv/rm"] --> KBP["KBPostgresBackend (fakes files over rows)"]
|
||||
KBP --> PR["path_resolver.py (computes fake /documents/ paths)"]
|
||||
AG --> MW["kb_persistence middleware (commit-at-end-of-turn → Postgres)"]
|
||||
MW --> DOCS[("documents + folders + chunks")]
|
||||
MW --> V1["DocumentVersion"]
|
||||
MW --> V2["DocumentRevision / FolderRevision + revert_service"]
|
||||
MW --> V3["AgentActionLog"]
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Decisions locked
|
||||
|
||||
- **Git = single source of truth** for all *indexed* KB content (agent/editor notes, uploads, indexable connectors — the `is_indexable` ones).
|
||||
- **Postgres = derived index only** (chunks + embeddings). It is a **cache**: rebuildable from Git via one `index_tree(workspace)`. Never authoritative. (As shipped, a rebuild upserts and prunes rather than wiping — document ids are in the Zero publication and must survive it; only chunk rows are replaced.)
|
||||
- **One-way derivation** (Git → Postgres). **Never** two-way sync (this is the Wiki.js anti-pattern we explicitly reject).
|
||||
- **Live connectors (Slack/Gmail) are untouched** — never stored/indexed, queried at chat time; entirely out of scope.
|
||||
- **Binary blobs stay in the existing blob store** (local/Azure). Git holds extracted markdown, not raw binaries (Git-LFS deferred).
|
||||
- **Agent tool interface is unchanged** (`ls/read/write/edit/mv/rm`). Only the backend behind the tools changes (Postgres-fake → real git working tree).
|
||||
- **History/undo = git log + `git revert`.** The three hand-rolled systems (`DocumentVersion`, `DocumentRevision`/`FolderRevision`+`revert_service`) are **deleted**.
|
||||
- **Engine = dulwich** (pure Python, deploy-friendly in Docker, real wire protocol for future remotes). Shell out to `git` only for heavy maintenance (`gc`/repack).
|
||||
- **Per-workspace write lock** is mandatory (git is single-writer) — a data-integrity boundary, not a feature.
|
||||
- **Rollout behind a feature flag**, per-workspace; no big-bang cutover.
|
||||
- **Ports & Adapters shape** ([ADR 0002](../../docs/adr/0002-knowledge-core-ports-and-adapters.md)): the KB is a framework-agnostic **core** (`KnowledgeStore`); deepagents is one **adapter**, not the core. See "Architecture shape" below.
|
||||
|
||||
## Architecture shape — Ports & Adapters (core + adapters)
|
||||
|
||||
The KB is a **framework-agnostic core** (`KnowledgeStore`) surrounded by adapters. deepagents is **one driving adapter**, not the core — because the same knowledge already has several consumers (chat agent, a KB REST API, the vector-store sync, later MCP / remote git). Coupling the core to deepagents would force every other consumer through an agent framework. Rationale + sources (Cockburn Hexagonal, git plumbing/porcelain, libgit2): [ADR 0002](../../docs/adr/0002-knowledge-core-ports-and-adapters.md).
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph ADAPTERS_IN["Driving adapters (world → core)"]
|
||||
DA["deepagents backend<br/>(Phase 2 — build now)"]
|
||||
API["KB REST API<br/>(deferred)"]
|
||||
MCP["MCP server<br/>(deferred)"]
|
||||
end
|
||||
CORE["KnowledgeStore<br/>(framework-agnostic core)"]
|
||||
ENGINE["VersionedContentEngine → GitContentEngine (dulwich)<br/>driven port — build now (Phase 1)"]
|
||||
SYNC["Vector-store sync / derived index<br/>driven consumer (Phase 4 — build now)"]
|
||||
REMOTE["Remote git (GitHub/GitLab)<br/>driven adapter (deferred)"]
|
||||
DA --> CORE
|
||||
API -.-> CORE
|
||||
MCP -.-> CORE
|
||||
CORE --> ENGINE
|
||||
CORE -->|commits| SYNC
|
||||
ENGINE -.-> REMOTE
|
||||
```
|
||||
|
||||
| Role | Consumer | v1? |
|
||||
|---|---|---|
|
||||
| Driving adapter | deepagents agent backend | ✅ build now (Phase 2) |
|
||||
| Driving adapter | KB REST API (Rohan's artifact API) | deferred — next adapter after the core |
|
||||
| Driving adapter | MCP server | deferred |
|
||||
| Driven port (infra) | storage engine (dulwich via `VersionedContentEngine`) | ✅ built (Phase 1) |
|
||||
| Driven consumer | vector-store sync / derived index | ✅ build now (Phase 4) |
|
||||
| Driven adapter | remote git (GitHub/GitLab) | deferred |
|
||||
|
||||
**YAGNI:** build only the **core + deepagents adapter + vector-store-sync consumer**. Shape the ports so REST/MCP/remote-git slot in later, but **do not build them now**. Grow the port surface on demand — capabilities (`ls`/`grep`/`glob`, version/diff verbs) live once in the core, and each adapter takes the subset it needs; no speculative methods, no per-consumer reimplementation.
|
||||
|
||||
## References we are borrowing from (not inventing)
|
||||
|
||||
Every decision traces to a proven source (full list + links in the ADR):
|
||||
|
||||
| Decision | Borrowed from |
|
||||
|---|---|
|
||||
| Git = truth, Postgres = rebuildable cache | **Fossil SCM** (`fossil rebuild`, production since 2007) |
|
||||
| Content in git, metadata/index in a DB | **Gollum** (GitHub/GitLab wikis), **kherad** |
|
||||
| Silent commit-per-save, hide git from users | **kherad** |
|
||||
| Embeddings keyed by blob SHA, incremental | **Coregit** + vector-index-as-cache best practices (LangChain RecordManager / LlamaIndex docstore) |
|
||||
| Python git engine | **dulwich** |
|
||||
| "Don't put a firehose in git" | "Git is not a database" critiques (validates keeping live connectors out) |
|
||||
| Reject two-way DB↔git sync | **Wiki.js** counter-example (requarks/wiki #7860 silent-sync bug) |
|
||||
|
||||
## Backend phases (active — this umbrella)
|
||||
|
||||
### Phase 0 — Shared contract [`subplan: 00c-shared-contract.md`]
|
||||
|
||||
> **READ FIRST.** Pins the five cross-phase contracts (repo/tree layout, read contract + citation model, lock, write path, index/Zero realities) grounded in the current code. Resolves what were per-phase "open questions". Two contracts were revised on 2026-07-28 after the adapter brainstorm: reads are **raw from the per-turn worktree** with one line-anchored citation pattern (C2), and in-turn writes live in a **per-turn worktree**, not the state overlay (C6).
|
||||
|
||||
### Phase 1 — Knowledge store core [`subplan: 01-git-storage-core.md`] ✅ implemented
|
||||
|
||||
> **DONE. Built first** — every later phase uses it.
|
||||
|
||||
- Added **dulwich**; a `KnowledgeStore` facade that opens/creates a **persistent working tree per workspace** on disk, nested under the shared blob-store volume (`{FILE_STORAGE_LOCAL_PATH}/knowledge_store/{workspace_id}`). Git lives behind the facade (`engines/git.py`).
|
||||
- API (SQL-transaction vocabulary, no git words; capabilities are verbs): `transaction(message, author)` scope yielding a `Transaction` with `write`/`remove`/`move` that records one atomic revision on clean exit; `read_as_of`, `list_revisions`, `list_changes`, `list_paths`, `get_current_revision`, `compute_content_id`. First use bootstraps the store — no init ceremony. Snapshot/batch is an engine detail. (Structure primitives, if any prove needed → Phase 2; undo/forward-restore → Phase 4.)
|
||||
- **Per-workspace Redis write lock** around commits (single-writer safety across all OS processes). `ponytail:` lock ceiling = one lock per commit; upgrade path = queue/worker.
|
||||
- Key files (new): `surfsense_backend/app/knowledge_store/` (`store`, `transaction`, `write_lock`, `store_path`, `settings`, `engines/{base,git}`). No agent wiring yet.
|
||||
- Tests — unit (`tests/unit/knowledge_store/`): engine behavior on temp repos, pure `Transaction` logic; integration (`tests/integration/knowledge_store/`, real Redis): write-lock semantics and the facade `transaction` end to end.
|
||||
|
||||
### Phase 2 — deepagents adapter over the core [`subplan: 02-git-working-tree-backend.md`]
|
||||
|
||||
> **SHIPPED (file-op path, 2026-07-28).** The **first driving adapter** ([ADR 0002](../../docs/adr/0002-knowledge-core-ports-and-adapters.md)) — deepagents talking to the core over the real git working tree, replacing the read-side fake. Remaining in-phase: the C2 `read_file` citation envelope (raw reads are line-numbered trivially; the normalizer's `:Lx-Ly` support, including the fail-closed strip for un-numbered entries, lands with it).
|
||||
|
||||
- `GitTreeBackend` serves `/documents/...` from the turn's **private working copy** (lazy open on the first KB tool call; copy id = `thread-{thread_id}`), implemented as one `MultiRootLocalFolderBackend` mount — no staging, no state overlay.
|
||||
- Working-copy lifecycle (`open`/`diff`/`discard`/`prune`) lives in the core behind the port; wired into `resolver.py` behind `KNOWLEDGE_STORE_ENABLED`; mutation tools route down the direct-op branches.
|
||||
- `path_resolver` + `KBPostgresBackend` retire for flagged workspaces (deleted at the Phase 5 cut).
|
||||
- Tests: lifecycle on temp repos, adapter behavior on real files, resolver gating.
|
||||
|
||||
### Phase 3 — Commit-per-turn write path [`subplan: 03-commit-write-path.md`]
|
||||
|
||||
> **SHIPPED (2026-07-29).** See the subplan's work items for the small as-built deviations.
|
||||
|
||||
- Persistence middleware `knowledge_store_persistence/` alongside `kb_persistence` (untouched until Phase 5 cut): end of turn → diff the working copy → one `transaction` → receipts → discard. Free-function commit body; the disconnect fallback in `event_loop.py` runs the identical routine (no state markers — the copy on disk is the pending state).
|
||||
- Model-generated commit subjects with deterministic fallback (`Thread:` trailer); the model seam is wired with the agent LLM until a weak-model role exists. Honest attribution: author = user, committer = agent (`knowledge_store/identities.py`).
|
||||
- Receipts created post-commit from `list_changes(revision)`, revision id as `external_id`; commit failure returns `failed` receipts and keeps the copy for next-turn recovery. Zero events move with Phase 4/6.
|
||||
- Editor saves route through `document_revision_recorder`; uploads and all connector indexers record at the `prepare_for_indexing` choke point, one revision per sync batch. Daily Celery beat janitor prunes abandoned copies.
|
||||
- Tests shipped: commit-turn scenarios against real git + Redis (net changes, no-op turns, contention recovery), message fallback, builder gating, recorder, janitor TTL.
|
||||
|
||||
### Phase 4 — Derived index + reindex [`subplan: 04-derived-index.md`]
|
||||
|
||||
> **SHIPPED (2026-07-30).** This is the **vector-store-sync driven consumer** ([ADR 0002](../../docs/adr/0002-knowledge-core-ports-and-adapters.md)): it subscribes to revisions one-way; the core has no knowledge of it. As-built record (with **Built as** deviation notes) in the subplan.
|
||||
|
||||
- `app/knowledge_store/index/converge.py`: one convergence body behind `index_changes` (paths moved since the stamp, fast queue) and `index_tree` (whole tree, upsert + prune, connectors queue), both converging to the store's current revision under a dedicated index lock; `workspaces.last_indexed_revision` is the stamp. Rows are adopted by ownership marker → NOTE hash → path; prune is keyed on the marker so connector rows are never touched.
|
||||
- Chunk line spans (`start_line`/`end_line`, migration 176) derived at the cache boundary by `attach_line_spans` — cached embeddings stay valid, no chunker-version bump, reconciler updates spans on moves (C2's consumer).
|
||||
- Writers enqueue indexing post-commit (`enqueue_index`), and an hourly capped sweep re-drives flipped workspaces whose stamp trails HEAD.
|
||||
- The three hand-rolled versioning systems are dead code for flagged workspaces (restore returns 409, editor reindex defers to the indexer); their **deletion (code + table drops) is Phase 5 cut time**.
|
||||
|
||||
### Phase 5 — Migration [`subplan: 05-migration.md`]
|
||||
|
||||
> **TOOLING SHIPPED (2026-07-30); fleet flips pending.** Seeder (`knowledge_store/migrate.py`), fleet runner (`scripts/migrate_knowledge_store.py`, seed → verify byte parity → flip only on pass), per-workspace flag (`workspaces.knowledge_store_enabled`, migration 175), daily drift monitor. No production workspace flipped yet; the cut-time deletion sweep (versioning code + table drops) runs after the fleet is verified.
|
||||
|
||||
- Export each existing workspace's Postgres documents/folders → an initial git repo (one seed commit), preserving `unique_identifier_hash` mapping.
|
||||
- **Adopt, don't rebuild** (amended 2026-07-29): parity = per-document **byte identity** vs Postgres, not `reindex()` — the seed copies bytes out of Postgres, so existing chunks/vectors already are its derived index; a full reindex is the 21-day-class job for zero information. The seed revision is the indexer's starting point, never incrementally indexed (else "every file added" = workspace-wide re-embed storm).
|
||||
- Rollback = keep Postgres content until the flagged workspace is verified; `reindex()` demoted to disaster recovery + a one-time pilot spot check.
|
||||
|
||||
### Phase 6 — Zero / real-time projection [`subplan: 06-zero-projection.md`]
|
||||
|
||||
> **PLANNED. The one net-new integration cost.** Depends on 03/04.
|
||||
|
||||
- The web UI is driven by Zero (Postgres logical replication, `zero_publication.py`). Git is not a real-time source, so add a **git → Postgres projection** that keeps the Zero-published `documents`/`folders` rows in sync after each commit (thin metadata rows, not content-authoritative).
|
||||
- Decide: reuse the indexer's post-commit hook to upsert Zero rows vs. a separate projector.
|
||||
- Tests: after a commit, the Zero-published rows reflect the new tree within the projection cycle; deleting a file removes its row.
|
||||
|
||||
## Sequencing (critical path vs. parallel)
|
||||
|
||||
- **Phase 0 first:** `00c` is a design agreement (no code) — sign off on the five contracts before starting `01`.
|
||||
- **Critical path:** `01 → 02 → 03` (storage → backend → write path). These deliver the core swap.
|
||||
- **Parallelizable:** `04` (indexer/reindex) can develop alongside `03`; both only need `01`'s `commit`/`log`.
|
||||
- **After core:** `05` (migration) then `06` (Zero projection). `06` is the only genuinely *new* subsystem (partly offsets the deletions) and must land before flagging a workspace whose UI must stay live.
|
||||
- Recommended: `01 → 02 → 03` (+`04` in parallel) → `06` → `05` → flip flag per workspace.
|
||||
|
||||
## Deferred — out of this umbrella
|
||||
|
||||
- **Connect-your-own-remote** (push/pull to user GitHub/GitLab/Gitea). Free later because the repo is real git.
|
||||
- **CRDT / Yjs real-time collaboration** (multi-writer). Keep single-writer for now.
|
||||
- **Review / merge workflows** (kherad's reviewer layer).
|
||||
- **Karpathy `raw/` + `wiki/` content model, contradiction-flagging, lint.**
|
||||
- **Graphiti / bi-temporal fact graph** (agent memory time-travel).
|
||||
- **Git-LFS for binaries** (blob store stays).
|
||||
- **Frontend/client umbrella** (version-history UI removal, any UX changes).
|
||||
|
||||
## Open items — resolved in Phase 0 ([`00c-shared-contract.md`](00c-shared-contract.md))
|
||||
|
||||
1. ~~Repo location & layout~~ → **persistent working tree at `{FILE_STORAGE_LOCAL_PATH}/knowledge_store/{workspace_id}`, layout = the full virtual path (`documents/...`), as shipped by the Phase-3 recorder and matched by the Phase-5 seeder** (C1).
|
||||
2. ~~Zero projection owner~~ → **folded into the Phase-4 post-commit indexer** (C5).
|
||||
3. **Binaries** — keep blob store (confirmed markdown/text-only in git); Git-LFS deferred.
|
||||
4. ~~Lock granularity~~ → **Redis lock, from v1** (deploy is multi-process: uvicorn + Celery) (C3).
|
||||
5. ~~`content_hash` vs blob SHA~~ → **different values (content_hash is workspace-salted); key reuse by blob SHA, keep content_hash through migration** (C5).
|
||||
6. **Migration cutover** — per-workspace flag flip after parity check; rollback = keep Postgres content until verified (Phase 5).
|
||||
|
||||
Still genuinely open (non-blocking): commit-message format, `gc`/repack scheduling, `reindex` observability.
|
||||
|
||||
## Resolved decisions log
|
||||
|
||||
- **(2026-07-24) PIVOT ADOPTED — Git as source of truth, Postgres as derived index.** Outcome of the KB maintenance investigation (ADR 0001). Git owns all *indexed* content; Postgres holds only chunks+embeddings and is rebuildable via `reindex()`. One-way derivation only (git→Postgres); two-way sync explicitly rejected (Wiki.js #7860). Live connectors (Slack/Gmail) unchanged and out of scope. Agent tool interface unchanged; only the backend behind it swaps. Three hand-rolled versioning systems to be deleted in favor of git history/`revert`. Engine = dulwich. Rollout behind a per-workspace feature flag.
|
||||
- **(2026-07-24) Connectors clarified — only `is_indexable` content enters git.** Document connectors (Notion, Drive, Obsidian) are indexed → they go into git. Live connectors (Slack, Gmail) are queried at chat time and never stored → they never touch git or Postgres chunks. (Corrects an earlier draft that carved *all* connectors out of git.)
|
||||
- **(2026-07-24) Borrowing, not inventing.** Architecture assembled from proven references (Fossil, Gollum, kherad, Coregit, dulwich, vector-cache best-practices); the only SurfSense-specific work is the adaptation glue. Wiki.js retained as the explicit counter-example.
|
||||
- **(2026-07-28) Ports & Adapters — deepagents is an adapter, not the core** ([ADR 0002](../../docs/adr/0002-knowledge-core-ports-and-adapters.md)). The KB is a framework-agnostic core (`KnowledgeStore`); consumers (deepagents, the KB REST API, the vector-store sync, later MCP/remote-git) are adapters at the edge. **YAGNI:** v1 builds only the core + the deepagents adapter (Phase 2) + the vector-store-sync consumer (Phase 4); REST/MCP/remote-git are named-but-deferred. Ports grow on demand. Borrowed from Cockburn (Hexagonal), git plumbing/porcelain, libgit2.
|
||||
|
||||
## Subplan index (backend)
|
||||
|
||||
| Phase | Subplan file | Status |
|
||||
|-------|--------------|--------|
|
||||
| 0 | `00c-shared-contract.md` | LOCKED — read first |
|
||||
| 1 | `01-git-storage-core.md` (core) | ✅ SHIPPED |
|
||||
| 2 | `02-git-working-tree-backend.md` (deepagents adapter) | ✅ SHIPPED (2026-07-28) — C2 `read_file` citation envelope still open |
|
||||
| 3 | `03-commit-write-path.md` | ✅ SHIPPED (2026-07-29) |
|
||||
| 4 | `04-derived-index.md` | ✅ SHIPPED (2026-07-30) |
|
||||
| 5 | `05-migration.md` | TOOLING SHIPPED (2026-07-30) — fleet flips + cut-time deletion pending |
|
||||
| 5a | `05a-seed-runbook.md` | operational runbook for the production seed + flip |
|
||||
| 6 | `06-zero-projection.md` | PLANNED |
|
||||
| — | `00b-diagrams.md` | companion flow diagrams |
|
||||
|
||||
Frontend & client subplans will be added under a separate umbrella later (see "Deferred").
|
||||
|
||||
## Appendix — current implementation file index (what each phase touches)
|
||||
|
||||
| Topic | Path |
|
||||
|---|---|
|
||||
| Virtual path resolver (retire) | `surfsense_backend/app/agents/chat/runtime/path_resolver.py` |
|
||||
| Virtual FS read backend (replace) | `.../filesystem/backends/kb_postgres.py` |
|
||||
| Backend resolver (rewire) | `.../filesystem/backends/resolver.py` |
|
||||
| Write commit middleware (repoint) | `.../main_agent/middleware/kb_persistence/middleware.py` |
|
||||
| Hybrid search (unchanged) | `.../shared/retrieval/hybrid_search.py` |
|
||||
| Chunk reconciliation (key by blob SHA) | `surfsense_backend/app/indexing_pipeline/chunk_reconciler.py` |
|
||||
| Indexing pipeline | `surfsense_backend/app/indexing_pipeline/indexing_pipeline_service.py` |
|
||||
| User version history (delete) | `surfsense_backend/app/utils/document_versioning.py` |
|
||||
| Agent revert (delete) | `surfsense_backend/app/services/revert_service.py` |
|
||||
| Zero publication (project into) | `surfsense_backend/app/zero_publication.py` |
|
||||
| ORM models | `surfsense_backend/app/db.py` |
|
||||
|
|
@ -1,172 +0,0 @@
|
|||
# Git-native KB — flow diagrams (end-to-end)
|
||||
|
||||
> Visual companion to [`00-umbrella-plan.md`](00-umbrella-plan.md).
|
||||
> Phase refs: `01` storage core · `02` working-tree backend · `03` commit write path · `04` derived index · `05` migration · `06` Zero projection.
|
||||
|
||||
## 1. The shape — one source of truth, one derived index
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph TRUTH["SOURCE OF TRUTH"]
|
||||
GIT["Git repo per workspace\n(commit per turn/save)"]
|
||||
end
|
||||
subgraph DERIVED["DERIVED (rebuildable cache)"]
|
||||
PG[("Postgres: chunks + embeddings")]
|
||||
end
|
||||
GIT -->|"one-way derivation (04)"| PG
|
||||
PG -. "reindex(workspace) rebuilds from git (04)" .-> GIT
|
||||
classDef t fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee;
|
||||
classDef d fill:#22314f,stroke:#5b7fbf,color:#e6edf7;
|
||||
class GIT t;
|
||||
class PG d;
|
||||
```
|
||||
|
||||
There is **no arrow from Postgres back into Git**. Postgres is disposable.
|
||||
|
||||
## 2. The system — hexagon: agnostic core, one adapter per consumer
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph DRIVERS["Drivers (who mutates / reads content)"]
|
||||
AG["Agent (deepagents tools)"]
|
||||
ED["Editor / REST API"]
|
||||
CN["Indexable connector sync"]
|
||||
end
|
||||
subgraph ADAPTERS["Adapters (consumer-specific glue)"]
|
||||
GTB["deepagents adapter (02)\nserves file ops on the turn's\nprivate working copy"]
|
||||
DIR["direct callers (03)\none transaction per save/sync"]
|
||||
end
|
||||
subgraph CORE["Knowledge store — agnostic core"]
|
||||
KS["Facade\ntransaction · read_as_of · list_revisions\nlist_changes · list_paths · working copies"]
|
||||
ENG["Versioned content engine\n(git today; swappable behind the port)"]
|
||||
end
|
||||
subgraph DRIVEN["Driven consumers (react to new revisions)"]
|
||||
IDX["Vector-store sync (04)\nlist_changes → re-chunk / re-embed"]
|
||||
ZP["Zero projection (06)"]
|
||||
end
|
||||
AG --> GTB --> KS
|
||||
ED --> DIR --> KS
|
||||
CN --> DIR
|
||||
KS --> ENG
|
||||
KS -->|"new revision"| IDX
|
||||
KS -->|"new revision"| ZP
|
||||
classDef core fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee;
|
||||
classDef edge fill:#22314f,stroke:#5b7fbf,color:#e6edf7;
|
||||
class KS,ENG core;
|
||||
class AG,ED,CN,GTB,DIR,IDX,ZP edge;
|
||||
```
|
||||
|
||||
Where each component lives (the code follows the dependency rule: the core
|
||||
never knows its consumers, so adapters sit with their consumer):
|
||||
|
||||
| Component | Location |
|
||||
| --- | --- |
|
||||
| Facade, transaction, write lock, layout | `app/knowledge_store/` |
|
||||
| Engine port + git engine | `app/knowledge_store/engines/` |
|
||||
| deepagents adapter (`GitTreeBackend`) + resolver | `app/agents/.../middleware/filesystem/backends/` |
|
||||
| End-of-turn commit middleware (03, pending) | `app/agents/.../middleware/` |
|
||||
|
||||
## 3. Turn lifecycle — git at the boundaries, plain files in between
|
||||
|
||||
Two locks, two different races:
|
||||
**threading lock** (process-local, in `GitContentEngine`) serializes parallel tool
|
||||
calls creating the same copy; **redis lock** (cross-process) serializes revision
|
||||
recording against other workers.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant T as Agent tool call
|
||||
participant A as GitTreeBackend (adapter)
|
||||
participant S as KnowledgeStore (facade)
|
||||
participant E as GitContentEngine (engine)
|
||||
T->>A: first KB op of the turn
|
||||
A->>S: open_working_copy("thread-{id}")
|
||||
S->>E: checkout current revision (threading lock)
|
||||
E-->>A: private copy path
|
||||
Note over T,A: rest of the turn: plain file ops on the copy\n(MultiRootLocalFolderBackend — no git involved)
|
||||
T->>A: end of turn (03, pending)
|
||||
A->>S: diff_working_copy → transaction
|
||||
S->>E: record one revision (redis write lock)
|
||||
S->>E: discard_working_copy
|
||||
Note over S,E: abandoned copies swept by janitor\n(prune_working_copies)
|
||||
```
|
||||
|
||||
## 4. Write path — everything indexed becomes a commit
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
AG["Agent edits (turn)"] --> WC["Private working copy per thread (02)"]
|
||||
ED["Editor save"] --> TXD["KnowledgeStore.transaction"]
|
||||
UP["Upload → extracted markdown"] --> TXD
|
||||
NOT["Indexable connector sync (Notion/Drive)"] --> TXD
|
||||
WC -->|"end of turn: diff → transaction (03)"| TXD
|
||||
TXD --> C["one revision recorded\n(redis write lock, 01)"]
|
||||
C --> IDX["Indexer: list_changes → changed blobs (04)"]
|
||||
IDX --> PG[("chunks + embeddings\n(embed keyed by content id)")]
|
||||
C --> ZP["Zero projection: upsert documents/folders rows (06)"]
|
||||
```
|
||||
|
||||
## 5. Read path — file ops vs. search hit different stores
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["Agent"] -->|"ls/read/write/edit/mv/rm"| B["GitTreeBackend → working copy (02)"]
|
||||
B --> GIT["Git (truth)"]
|
||||
A -->|"semantic search"| S["hybrid_search (unchanged)"]
|
||||
S --> PG[("Postgres chunks + embeddings")]
|
||||
```
|
||||
|
||||
## 6. Live connectors — never stored (out of scope)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Q["Chat query"] --> LC["Slack / Gmail (live)"]
|
||||
LC -->|"fetched at chat time"| ANS["used in the answer"]
|
||||
LC -.->|"never"| GIT["Git"]
|
||||
LC -.->|"never"| PG[("Postgres chunks")]
|
||||
```
|
||||
|
||||
## 7. History / undo — git replaces the three hand-rolled systems
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph OLD["BEFORE (deleted)"]
|
||||
V1["DocumentVersion"]
|
||||
V2["DocumentRevision / FolderRevision + revert_service"]
|
||||
V3["AgentActionLog (audit)"]
|
||||
end
|
||||
subgraph NEW["AFTER"]
|
||||
L["git log / diff (history)"]
|
||||
R["git revert (undo)"]
|
||||
BL["git blame (attribution)"]
|
||||
end
|
||||
OLD -->|"replaced by (04)"| NEW
|
||||
```
|
||||
|
||||
## 8. Migration (05) — Postgres KB → seed git repo
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant M as Migrator (per workspace, flagged)
|
||||
participant PG as Postgres (existing docs/folders)
|
||||
participant GIT as New git repo
|
||||
participant IDX as reindex(workspace)
|
||||
M->>PG: read documents + folders (preserve unique_identifier_hash)
|
||||
M->>GIT: write files + one seed commit
|
||||
M->>IDX: rebuild chunks/embeddings from git HEAD
|
||||
IDX-->>M: chunk set
|
||||
M->>M: verify search parity vs pre-migration, then flip flag
|
||||
Note over M,GIT: Postgres content kept until verified (rollback window).
|
||||
```
|
||||
|
||||
## 9. Reindex (04) — the safety net (Fossil `rebuild`)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
GIT["Git HEAD (truth)"] --> RB["reindex(workspace)"]
|
||||
RB --> WIPE["wipe chunks + embeddings"]
|
||||
WIPE --> REBUILD["re-chunk + re-embed all files\n(reuse cache by blob SHA)"]
|
||||
REBUILD --> PG[("Postgres index rebuilt")]
|
||||
```
|
||||
|
|
@ -1,127 +0,0 @@
|
|||
# Phase 0 — Shared contract (resolve before building)
|
||||
|
||||
> The load-bearing decisions every later phase depends on. These were left as
|
||||
> per-phase "open questions" in the first draft; they are **shared**, so they live here
|
||||
> once and the phases reference them. Umbrella: [`00-umbrella-plan.md`](00-umbrella-plan.md).
|
||||
> Grounded in the current code — symbols and paths below are real.
|
||||
|
||||
This doc is the answer to "is it clear enough for a dev to implement?" — it pins the
|
||||
contracts that phases 1–6 assume. Read it first.
|
||||
|
||||
---
|
||||
|
||||
## C1 — Repo location & tree layout (Phase 1, 5)
|
||||
|
||||
**Decided.**
|
||||
|
||||
- **One repo per workspace** at `{FILE_STORAGE_LOCAL_PATH}/knowledge_store/{workspace_id}` (nested **under** the shared blob-store volume, so every OS process sees the same history). Persistent working tree per workspace (not bare + ephemeral checkout) — simpler, and the per-workspace lock (C3) makes a single live checkout safe.
|
||||
- **Tree layout mirrors today's virtual paths in full.** A doc at virtual `/documents/<folder>/<title>.xml` lives in git at `documents/<folder>/<title>.xml` — the leading slash stripped, the `documents/` root kept. The root is deliberate: it reserves the repo's top level for future sibling roots (e.g. `.cache/`, `artifacts/`) without ever moving existing paths. (Amended 2026-07-29: originally "minus the root"; the Phase-3 recorder shipped keeping it and the Phase-5 seeder matches.)
|
||||
- **Reuse the existing filename rules** — do **not** reinvent them. `safe_filename`, `safe_folder_segment`, and the ` (<doc_id>).xml` collision suffix all come from `app/agents/chat/runtime/path_resolver.py`. Keep the `.xml` extension in v1 (changing to `.md` would break `unique_identifier_hash`, which is computed from the virtual path — see C2).
|
||||
- **Git stores the source text** (the agent's note = `Document.source_markdown`/`content`), one file per document. Not the rendered XML view (that's derived — see C2). Not binaries (stay in the blob store).
|
||||
- **Identity mapping preserved.** `unique_identifier_hash = generate_unique_identifier_hash(DocumentType.NOTE, virtual_path, workspace_id)` (`app/utils/document_converters.py`) stays the stable doc identity across the git↔Postgres boundary and for connector re-sync. Store it in the derived `documents` row (as today); the git path is the human-facing identity.
|
||||
|
||||
## C2 — Read contract: raw reads + one citation pattern (Phase 2, 4)
|
||||
|
||||
**Decided 2026-07-28 — supersedes this contract's earlier rendered-from-chunks model.** Chunk-render preserved four defects: the agent read a citation-polluted artifact (markers baked into cached file bytes), citations pointed at chunk ids that dangle on every reindex, granularity was the chunker's whim, and reads depended on the derived index. Rohan flagged citation quality; this redesigns it instead of porting it.
|
||||
|
||||
**Reads are raw.** `read_file` returns the real file from the turn's worktree (C6), line-numbered, honest `offset`/`limit`. Citation markers never enter file bytes — the handle rides in the envelope only, so edits and commits operate on clean content.
|
||||
|
||||
**One citation pattern through both doors (search excerpts and full reads).** All KB content reaches the agent as true-document-line-numbered text inside the same envelope, handle in the opening tag:
|
||||
|
||||
```text
|
||||
<document title="…" path="…" cite="[3]" view="excerpts|full" revision="…">
|
||||
340 ## Refunds — annual plans
|
||||
341 Annual subscriptions may be refunded within 30 days…
|
||||
</document>
|
||||
```
|
||||
|
||||
One agent rule, zero per-tool special cases: cite `[n]`, narrowed to the evidence lines seen — `[3:L341-L342]`. Multiple chunks of one document collapse into one envelope, one handle.
|
||||
|
||||
**Citation currency = `(path, revision, evidence lines)`.**
|
||||
|
||||
- Registry entries are per-document and **self-contained**: `{path, revision, title}`, snapshotted at render time, persisted with the chat. Never a chunk id — rebuilds can't dangle what they can't reach.
|
||||
- Evidence lines come from the **agent's emitted qualifier**; the normalizer parses the optional `:Lx-Ly` suffix, clamps it to the document length at that revision, and strips it when the entry was rendered un-numbered (see spans paragraph below).
|
||||
- Resolution touches the chat's registry + git only: `read_as_of(revision, path)` → open at revision, highlight lines; the hover snippet is read from git at those exact lines. Chunk rows are never consulted after render.
|
||||
|
||||
**Chunk spans: stored at cut time, never migration-backfilled (amended 2026-07-29, second pass).** Phase 4's indexer writes `start_line`/`end_line` as it cuts (chonkie already returns exact offsets; verified `chunk.text == text[start:end]`). The columns are nullable — an instant, metadata-only ALTER; **what killed PR #1523 and the measured ~21-day chunk backfill was the mandatory table rewrite, not the column**, so no migration step ever fills them. Render-time derivation was considered and rejected: matching chunk text against the blob per query can silently pick the wrong occurrence of repeated text and puts git I/O in the search hot path.
|
||||
|
||||
**Legacy chunks (`NULL` spans) render un-numbered and cite at document level — fail-closed.** The mixed period (some excerpts numbered, some not) is a hallucination surface: the agent could invent `Lx-Ly` for un-numbered excerpts. The guard is mechanical, not prompt-hope: registry entries record whether they were rendered with numbers, and the normalizer **strips a `:Lx-Ly` qualifier whose entry was rendered un-numbered** (alongside the existing clamp). Correctness never depends on spans; they are progressive enhancement.
|
||||
|
||||
**Convergence = a deadline-free daily fill job (operational, not a migration gate).** Runs on flipped workspaces only. CPU-only, no embeddings: per document, match the stored chunk texts **against the git blob** — the text spans will actually resolve against (`read_as_of`), which post-flip *is* the source of truth; `source_markdown` is by then a derived projection and is never consulted. The match is a cursor scan in position order over *all* the document's chunks (repeated text disambiguates by alignment), updating only the span columns. Walks workspaces **most-used first**, batched/throttled, pausable at any point, until the NULL-span count reaches zero; new chunks always get spans at cut, so the backlog only shrinks.
|
||||
|
||||
One fail-closed guard, mandatory: **all-or-nothing per document** — every chunk must match at ascending start offsets or the document is skipped entirely. Every drift scenario falls out correctly from it: blob ahead of chunks (edit/index lag) → no alignment → skip, the next index re-chunks with spans anyway; blob behind Postgres (coexistence git-record failure) → no alignment → skip, correctly, since spans into a stale blob would resolve wrong; untouched-since-seed (the common case) → parity made blob = the chunks' birth text → full alignment, correct spans. Skipped and unmatched chunks stay NULL and keep document-level citations. No path yields confidently wrong line numbers.
|
||||
|
||||
`ponytail:` evidence lines are agent-emitted (the Claude Code / Cursor precedent); a sloppy model can mis-cite. Clamp-only in v1; upgrade path = validate cited lines against what was actually shown in the turn.
|
||||
|
||||
Structure (`ls`/`glob`/`grep`/`list_tree`) comes from the worktree; derivation stays one-way (git → Postgres, no two-way sync).
|
||||
|
||||
## C3 — Concurrency: Redis lock, from v1 (Phase 1, 3)
|
||||
|
||||
**Decided — this corrects subplan 01's "in-process asyncio lock (v1)".**
|
||||
|
||||
The backend runs as **multiple OS processes**: the API (`python main.py`, uvicorn, default **4** workers per `docker/.env.example`) **and** Celery workers (`SERVICE_ROLE=worker`, autoscale **2–10**), sometimes in one container (`SERVICE_ROLE=all`). An in-process `asyncio.Lock` **cannot** serialize writes across them — it would give false single-writer safety.
|
||||
|
||||
- **Use a Redis lock** keyed `knowledge_store:write_lock:{workspace_id}` (token-owned release, TTL + queue-then-fail, fail-if-down). Redis is already a hard dependency (Celery broker/result/app cache), so this adds no infra.
|
||||
- `ponytail:` v1 ceiling = one Redis lock per workspace held for the duration of a commit; upgrade path = a per-workspace write queue/worker if contention shows up.
|
||||
- Alternative if Redis is ever removed: a Postgres advisory lock (`pg_advisory_xact_lock(hashtext('knowledge_store:'||workspace_id))`). Redis is the default.
|
||||
|
||||
## C4 — Write path: repoint `commit_staged_filesystem_state` (Phase 3, 4)
|
||||
|
||||
**Decided — updated 2026-07-28 for the worktree model (C6).** The end-of-turn commit body is `commit_staged_filesystem_state(...)` in `.../kb_persistence/middleware.py` (called by `aafter_agent` and the stream-task fallback). For flagged workspaces, repoint it from "drain staged state keys to Postgres" to "**record the turn's worktree diff** as one revision" (C6: `porcelain.status` → change set → `store.transaction()`).
|
||||
|
||||
- **The staged-op state keys and their ordered drain are legacy-path only.** With the worktree, the tree already holds the netted outcome — moves are applied file ops, write-then-`rm` nets to nothing — so the key list (`files`, `staged_dirs`, `pending_moves`, `pending_deletes`, `pending_dir_deletes`, `dirty_paths`, `doc_id_by_path`, …) and the five-step ordering survive only on the unflagged Postgres path until migration deletes them (with `_pending_filesystem_view`, C6).
|
||||
- **Author** = `created_by_id` (the acting user id passed into the middleware); use `agent` for autonomous writes. **Message** summarizes the turn's ops.
|
||||
- **Coupling with Phase 4:** the `DocumentRevision`/`FolderRevision` snapshot logic (gated by `flags.enable_action_log`) lives *inside* this function. Deleting those systems (Phase 4) removes that snapshot code from here — sequence Phase 3's rewrite and Phase 4's deletion together for flagged workspaces.
|
||||
- **Emit the commit SHA** into state/event for Phase 4 (index) and Phase 6 (project). Keep the `dispatch_custom_event` calls (`document_created/updated/deleted`, `folder_deleted`) — the UI depends on them (C5).
|
||||
|
||||
## C5 — Index + Zero projection realities (Phase 4, 6)
|
||||
|
||||
**Decided / corrected.**
|
||||
|
||||
- **An embedding cache exists — extend it, don't build one.** *(Corrected 2026-07-29; the original "no cache exists" claim overlooked `indexing_pipeline/cache/`, in the tree since 2026-06-12.)* `build_chunk_embeddings` serves a document's full chunk+vector set keyed by `(markdown_sha256, embedding_model, dim, chunker_kind, chunker_version)`. `markdown_sha256` is a pure, unsalted content hash — the "content id" property Phase 4 needs — so identical bytes already re-embed nothing. A separate blob-SHA reuse layer is unnecessary; `chunk_reconciler.reconcile` remains the row-level in-place reuse on top. **Do not warm this cache from legacy `Chunk` rows**: they may predate the current `chunker_version`, and writing their boundaries under today's key would poison it.
|
||||
- **`content_hash` ≠ git blob SHA.** `generate_content_hash(content, workspace_id)` is **workspace-salted**; a git blob SHA is content-only and unsalted. They are different values — you cannot just alias one onto the other. Recommend: key embedding reuse by **blob SHA**; keep `content_hash` for existing document-level checks through migration, drop later if redundant.
|
||||
- **Real-time UI has two channels, both must survive.** (1) Zero logical replication of the `documents`/`folders` rows (`app/zero_publication.py`); (2) `dispatch_custom_event` SSE from the commit path. Phase 6's git→Postgres projection must **upsert/delete the `documents`/`folders` rows** (so Zero streams them) **and** keep emitting the same custom events. Simplest owner: the Phase-4 post-commit pass does both (index + project) in one shot.
|
||||
|
||||
## C6 — In-turn writes live in a per-turn private worktree (Phase 2, 3)
|
||||
|
||||
**Decided 2026-07-28 — supersedes this contract's earlier state-overlay model.** The overlay was Postgres-backend debt, not a framework requirement (deepagents' `FilesystemBackend` writes directly to disk). The worktree deletes `_pending_filesystem_view` and the six staged-state keys instead of porting them.
|
||||
|
||||
Each turn that touches the KB gets a **private detached git worktree** of the workspace repo, checked out at the current revision:
|
||||
|
||||
- **Created lazily** on the turn's first KB tool call (read or write); turns that never touch the KB pay nothing. Measured: ~90 ms checkout for a 500-doc workspace, once per turn.
|
||||
- **One tree serves the whole turn.** Every `read`/`write`/`edit`/`ls`/`glob`/`grep` in the turn (sub-agents included) is a plain file op on it — one code path, read-your-own-writes by construction, no overlay, no merge logic.
|
||||
- **Abort/crash = delete the directory.** An age-based janitor prunes orphans. Nothing uncommitted survives.
|
||||
- **Commit = diff, not snapshot.** At end of turn, `porcelain.status(worktree)` (dulwich; the detached HEAD *is* the base revision) yields the touched paths → mapped to a `writes`/`removes` change set → recorded via `store.transaction()` under the Redis lock (C3), on top of the current head. Committing only the diff means a parallel turn's already-committed work on untouched files is never reverted.
|
||||
- **Same-file overlap: last-writer-wins, with history.** No merge machinery in v1; the overwritten version stays reachable in the prior revision. `ponytail:` ceiling = no three-way merge / conflict surfacing; upgrade path = git's own three-way merge if concurrent same-file edits become real.
|
||||
- **Moves are stored as delete+add** — identical to git's own rename storage; no fidelity lost.
|
||||
|
||||
`ponytail:` known trade: a mid-turn checkpoint fork/replay does not restore uncommitted worktree files (state staging would have). Accepted — SurfSense has no mid-turn fork feature; revisit only if one appears.
|
||||
|
||||
## C7 — Migration seed is adopted, never incrementally indexed (Phase 4, 5)
|
||||
|
||||
**Decided 2026-07-29.** The Phase-5 seed commit copies bytes *out of Postgres*, so the existing `chunks` rows and vectors already are its derived index — byte parity is the proof. To `index_revision`, though, the seed looks like "every file added"; feeding it through would re-embed the whole workspace (the ~21-day job class that killed the chunk-column backfill and PR #1523).
|
||||
|
||||
- The seeder records the seed revision as the indexer's **starting point**; incremental indexing begins with the first post-seed revision.
|
||||
- `reindex(workspace)` is disaster recovery + a one-time pilot spot check — never a per-workspace migration gate.
|
||||
- Migrated chunks converge to the current chunker **lazily, on edit** (reconciler + embedding cache bound the cost). Never eagerly re-chunk, and never warm the cache from legacy rows (C5). The C2 span fill job is the one exception — it updates span columns only (no re-chunk of rows, no embeddings) and gates nothing.
|
||||
|
||||
---
|
||||
|
||||
## What stays exactly as-is (do not touch)
|
||||
|
||||
- Hybrid search (`.../shared/retrieval/hybrid_search.py`) — reads the same `chunks` table.
|
||||
- Live connectors (Slack/Gmail) — never stored/indexed.
|
||||
- Desktop-local backend (`MultiRootLocalFolderBackend`) — real filesystem already.
|
||||
- Blob store for binaries.
|
||||
|
||||
## Resolved-here index (was per-phase "open questions")
|
||||
|
||||
| Was open in | Now decided in |
|
||||
|---|---|
|
||||
| 01 lock granularity; repo layout; repo root | C3, C1 |
|
||||
| 02 read rendering; glob/grep source | C2 |
|
||||
| 02 in-turn write visibility; turn isolation (worktree) | C6 |
|
||||
| 03 author identity; staged-op keys | C4 |
|
||||
| 04 content_hash vs blob SHA; cache location | C5 |
|
||||
| 05 parity gate; seed vs incremental indexer | C7 |
|
||||
| 06 projection owner; consistency | C5 |
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
# Phase 1 — Knowledge store core ✅ implemented
|
||||
|
||||
> Build first; every later phase uses this. Umbrella: [`00-umbrella-plan.md`](00-umbrella-plan.md).
|
||||
> No agent wiring here — this phase is a standalone, tested versioned-storage service + per-workspace write lock.
|
||||
|
||||
## Objective
|
||||
|
||||
A `KnowledgeStore` facade that owns one versioned history per workspace and exposes the small, engine-agnostic set of primitives the rest of the pivot needs, with single-writer safety. Engine = **dulwich** (pure Python; no system `git` dependency in the container; real wire protocol so future "bring your own remote" is free), confined behind the facade.
|
||||
|
||||
## Locked model
|
||||
|
||||
- **One store per workspace**, persistent working tree, at `{FILE_STORAGE_LOCAL_PATH}/knowledge_store/{workspace_id}` (nested **under** the shared blob-store volume so every process sees the same history; see [`00c-shared-contract.md`](00c-shared-contract.md) C1 for filename rules — reuse `path_resolver`'s `safe_filename`/`safe_folder_segment`, keep `.xml`).
|
||||
- **Markdown/text only** in git; binaries stay in the blob store (Phase-agnostic; see umbrella).
|
||||
- **Single-writer per store via a Redis lock** keyed `knowledge_store:write_lock:{workspace_id}` — mandatory from v1, **not** an in-process `asyncio.Lock`. The backend runs as multiple OS processes (uvicorn workers + Celery workers), so an in-process lock gives false safety; Redis is already deployed. Token-owned release, 30s TTL, 10s queue-then-fail, and fail-if-Redis-down (a write never proceeds unserialized). `ponytail:` v1 ceiling = one Redis lock held per commit; upgrade path = per-workspace write queue. Full rationale: [`00c-shared-contract.md`](00c-shared-contract.md) C3.
|
||||
- **dulwich for the hot path**, shell out to `git gc`/repack only for periodic maintenance (not in v1).
|
||||
|
||||
## What shipped
|
||||
|
||||
1. `dulwich` added to `surfsense_backend` deps.
|
||||
2. Package `app/knowledge_store/`:
|
||||
- `settings.py` — `load_knowledge_store_settings()` (enabled flag + root, from central config).
|
||||
- `store_path.py` — `workspace_store_path(workspace_id)`: the sole owner of on-disk layout.
|
||||
- `write_lock.py` — `workspace_write_lock(workspace_id)` async context manager over the Redis lock (C3), with explicit TTL/wait constants and `KnowledgeStoreLockError`.
|
||||
- `transaction.py` — `Transaction`: the unit-of-work verbs (`write`/`remove`/`move`) and their resolution into concrete writes/removes (`resolve`).
|
||||
- `store.py` — `KnowledgeStore` async facade (runs the sync engine via `asyncio.to_thread`; reads are lock-free, writes serialized). Public surface — **intent verbs, no git vocabulary**:
|
||||
- First use bootstraps the store — no init ceremony; queries on a virgin store answer empty.
|
||||
- `transaction(message, author)` — an atomic unit-of-work scope (SQL `BEGIN`/`COMMIT` shape, Django `transaction.atomic()` precedent) yielding a `Transaction` with verbs `write(path, content)` / `remove(path)` / `move(src, dst)`. On clean exit it records **exactly one revision** under the write lock (`tx.revision` = the new id, `None` if nothing changed); on exception it records nothing. Whether that revision touches one file or fifty is an engine detail.
|
||||
- `read_as_of(revision, path)` (temporal read, SQL/Datomic "as of"), `list_revisions(path=None, limit=None)`, `get_current_revision()` (a revision is always a whole-workspace snapshot).
|
||||
- Driven-consumer reads (Phase 4's inputs): `list_changes(revision)` — paths added/modified/removed vs the parent, with content ids; `list_paths(revision)` — full enumeration for `reindex`.
|
||||
- `compute_content_id(data)` — git blob SHA (content-addressed id, consumed by Phase 4).
|
||||
- `engines/base.py` — `VersionedContentEngine` contract (**engine boundary**: `record(writes, removes)`, `read`, `read_as_of`, `list_revisions`, `list_changes`, `list_paths`, `get_current_revision`, `compute_content_id`) + `Revision`/`Change`/`TrackedPath`. `engines/git.py` — `GitContentEngine` (all dulwich mechanics; the swappable engine seam — git vocabulary starts here, not in the port). The verb→snapshot translation lives in the facade, so the batch never surfaces in the API.
|
||||
3. Config flags `KNOWLEDGE_STORE_ENABLED` (off by default) + `KNOWLEDGE_STORE_ROOT`.
|
||||
|
||||
## Tests
|
||||
|
||||
Unit (`tests/unit/knowledge_store/`) covers what runs locally for real; anything whose correctness depends on Redis is integration (`tests/integration/knowledge_store/`, real Redis).
|
||||
|
||||
- **Engine, unit** (`GitContentEngine` on temp repos): first use bootstraps the store; a mixed write+modify+delete lands in one revision; no-op record returns `None`; removing an untracked path is tolerated; `list_revisions` newest-first, path-scoped, honors `limit`; `list_changes` reports added/modified/removed with content ids; `list_paths` reflects the given revision; revisions carry author + tz-aware timestamp; `compute_content_id` equals real `git hash-object`.
|
||||
- **Transaction, unit** (pure logic): verbs net into one change set; move resolves from staged or committed content; moving a missing path raises.
|
||||
- **Facade, integration** (`KnowledgeStore.transaction` + real Redis): one scope records one revision; an exception inside the scope records nothing; a transaction fails cleanly while another writer holds the workspace.
|
||||
- **Write lock, integration** (real Redis): one writer per workspace; workspaces don't contend; released on scope exit and on exception.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Agent/backend wiring → Phase 2. Commit-on-turn → Phase 3. Indexing → Phase 4.
|
||||
- Structure primitives (`list_tree`/glob/grep) → added in Phase 2. Undo/forward-restore → Phase 4 (v1 is `read_as_of` + `history` only).
|
||||
- Remotes (push/pull), Git-LFS, `gc`/repack scheduling — deferred (umbrella).
|
||||
|
||||
## Resolved (see [`00c-shared-contract.md`](00c-shared-contract.md))
|
||||
|
||||
- **Lock:** Redis lock, from v1 (C3) — deploy topology is multi-process, so in-process locks are out.
|
||||
- **Repo model:** persistent working tree per workspace (C1).
|
||||
- **Repo root:** `{FILE_STORAGE_LOCAL_PATH}/knowledge_store/{workspace_id}` (C1); backup/retention folds into existing blob-store backup.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. `gc`/repack scheduling threshold (deferred to a later ops pass, not v1).
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
# Phase 2 — deepagents adapter over the core
|
||||
|
||||
> Build after Phase 1. Umbrella: [`00-umbrella-plan.md`](00-umbrella-plan.md). Shape: [ADR 0002](../../docs/adr/0002-knowledge-core-ports-and-adapters.md).
|
||||
> The **first driving adapter**: deepagents talking to the framework-agnostic core (`KnowledgeStore`) over the **real git working tree**, replacing the read-side fake (`KBPostgresBackend`). Agent tools are unchanged.
|
||||
>
|
||||
> **Status: file-op path implemented (2026-07-28); flip-safe as of 2026-07-30.** Working-copy lifecycle in the core, `GitTreeBackend` adapter, resolver + tool routing behind `KNOWLEDGE_STORE_ENABLED`. The C2 citation envelope on `read_file` is **deferred to its own post-flip PR** — it is not a flip blocker; what was, the `read_file` description promising a citation envelope the git path never renders, is fixed (see work item 4).
|
||||
|
||||
## Objective
|
||||
|
||||
Give the agent real file ops (`ls/glob/grep/read/write/edit`) on a **per-turn private worktree** instead of `path_resolver` + a DB folder walk + the `runtime.state` overlay — as a **thin deepagents adapter**, not a bespoke filesystem reimplementation. The adapter is deepagents' own `FilesystemBackend` pointed at the turn's worktree; the core is invoked once, at the end-of-turn commit.
|
||||
|
||||
## Locked model
|
||||
|
||||
- **This is an adapter, not the core.** The core (`KnowledgeStore`) stays framework-agnostic (imports no deepagents). Only this adapter knows about `BackendProtocol`. It adds no storage logic of its own.
|
||||
- **In-turn writes live in a per-turn private worktree** (decided 2026-07-28; C6 records this model). Git's own answer to parallel writers: one working tree per writer, never a shared checkout. The worktree is **created lazily on the turn's first KB tool call** (read or write; measured ~90 ms checkout for 500 docs, once per turn) and serves **both reads and writes** for the whole turn — one code path, read-your-own-writes by construction, **no state overlay, no merge logic**. Abort/crash = delete the worktree; a janitor removes orphans.
|
||||
- **End-of-turn commit = diff, not snapshot.** `porcelain.status(worktree)` (the detached HEAD *is* the base revision) → touched paths → `writes`/`removes` change set → `store.transaction()` under the Redis lock, on top of the current head. Parallel turns' committed work on untouched files is never reverted; same-file overlap is per-file last-writer-wins **with the loser preserved in history**. `ponytail:` ceilings = no three-way merge (upgrade path: git's own), and a mid-turn checkpoint fork/replay does not restore uncommitted worktree files — accepted, no such feature exists.
|
||||
- **Reuse the existing direct-disk backend; don't reimplement the framework.** Implemented as SurfSense's own `MultiRootLocalFolderBackend` with one mount, `("documents", <working copy>)` — it already ships the whole tool surface (`als_info`/`aread`/`awrite`/`aedit`/`aglob_info`/`agrep_raw`/`alist_tree`/`amove`/`adelete_file`/`armdir`) plus root-confined path resolution, which deepagents' `FilesystemBackend` + `CompositeBackend` would have covered only partially (no move/delete/rmdir/list_tree).
|
||||
- **Same tool interface.** Upstream `WriteResult`/`EditResult`/`FileInfo`/`GrepMatch` shapes — no extra fields.
|
||||
- **`path_resolver` path computation retires for flagged workspaces** — folder walk + collision suffixing → real repo paths (filename rules still reused, C1). **The repo tree keeps the `documents/` root** (C1 as shipped: repo path = virtual path minus only the leading slash), leaving the top level free for future sibling roots (`.cache/`, `artifacts/`). `GitTreeBackend` mounts the working copy's `documents/` subtree under the mount name `documents`, so agent writes land under the prefix; the editor/upload recorder and the migration seeder produce the same paths via `path_resolver.to_store_path`.
|
||||
- Selected via the resolver behind `KNOWLEDGE_STORE_ENABLED`.
|
||||
|
||||
## Citation model — decided 2026-07-28 (full contract: C2)
|
||||
|
||||
Raw reads from the worktree; one citation pattern through both doors. Every KB surface (search excerpt or full read) renders as true-document-line-numbered text in one envelope with the handle in the opening tag (`cite="[n]"`); the agent cites `[n:Lx-Ly]` using the line numbers it sees. Registry entries are self-contained `{path, revision, title}` — no chunk ids, markers never enter file bytes. Retires `aload_document`/`render_full_document`'s chunk-render read path for flagged workspaces.
|
||||
|
||||
## Work items
|
||||
|
||||
1. ✅ **Working-copy lifecycle in the core** (git vocabulary stays behind the port): `open_working_copy` / `diff_working_copy` / `discard_working_copy` / `prune_working_copies` on the port, facade, and engine (`dulwich.worktree`). Copies live at `{root}/.working_copies/{workspace_id}/{copy_id}` (`store_path.py` owns the layout). An **empty store yields a bare directory** (git cannot worktree an unborn HEAD); its diff walks the tree.
|
||||
2. ✅ New `.../filesystem/backends/git_tree.py` — `GitTreeBackend`: lazy mount of the turn's working copy, **opened on the first KB tool call**. Copy id = `thread-{root thread}` (langgraph serializes turns per thread; a copy left by a crashed turn is committed with the thread's next turn — recovery semantics; abandoned copies are janitored). **The copy is scoped to the turn, not to the actor**: subagents run under a namespaced `{parent}::task:{tool_call_id}` thread id (one segment per nesting level), so `thread_working_copy_id` resolves the root segment and every actor in the turn — orchestrator and nested subagents alike — shares one copy. That is also what keeps one turn to one revision, which the receipts and citation revisions assume. `open_working_copy` serializes its check-then-create behind a module lock, so parallel subagents reopen rather than race.
|
||||
3. ✅ Wired into `resolver.py::build_backend_resolver` behind `KNOWLEDGE_STORE_ENABLED`; mutation tools (`rm`/`rmdir`/`move_file`/`mkdir`) route `GitTreeBackend` down the existing direct-op branches instead of cloud state-staging — as do `write_file`/`edit_file`, which stage as a side effect of a successful write and were missed in the first pass (see the canary findings below). Root-cause fix along the way: `mkdir` was a silent no-op on every direct backend while `write` refuses missing parents — added real `mkdir` to `LocalFolderBackend` (+ multi-root passthrough) and made the tool surface backend errors.
|
||||
4. ⏳ `read_file` C2 citation envelope + normalizer `[n:Lx-Ly]` support (today the flagged path returns the raw line-numbered read with no envelope).
|
||||
|
||||
**Flip-safety split off and shipped (2026-07-30); the envelope itself is deferred to its own post-flip PR.** The envelope is not a flip blocker — search citations are unaffected, since `search_knowledge_base` reads the `chunks` rows the indexer writes and resolves through `/documents/by-chunk/{id}` unchanged. The *description* was: `select_description` ignored its `mode` argument and told every mode that reads come back as `<document … view="full">` with `[n]`-labelled passages, adding "cite the same `[n]` you would use from `search_knowledge_base`". Only `KBPostgresBackend` renders that envelope, so on a flipped workspace the model was promised labels it never sees while holding search ordinals in context — a mis-citation surface (right ordinal, wrong source), which is worse than the missing citation it looks like. Now split by read format rather than by cloud-vs-desktop: the envelope text for cloud-on-Postgres, and a raw-file text that forbids reusing a search ordinal for everything else. This **also fixed desktop-local**, which had carried the same wrong description since before the flag existed. `knowledge_store_enabled` threads `stack.py` → subagent deps → `build_filesystem_mw` → middleware, and already keys the compiled-graph cache.
|
||||
|
||||
Remaining, as a separate PR, in four slices: (1) the envelope + registry entries carrying `{path, revision, title, numbered}` + normalizer `[n:Lx-Ly]` parsing with a fail-closed strip; (2) the line range carried through to a new payload, a `read_as_of` resolution endpoint, and the frontend panel — **this slice is cross-stack**, since `surfsense_web` resolves citations only by numeric chunk id today and C2's currency is `(path, revision, lines)`; (3) search excerpts through the same envelope using the stored spans; (4) the span-fill job for legacy `NULL`-span chunks. Slices 3 and 4 *depend on* the flip: spans exist only where the git indexer has run, and C2 defines the fill job as flipped-only, matching against the git blob because post-flip that is the source of truth.
|
||||
5. ✅ Janitor scheduling — shipped with Phase 3: daily Celery beat task via `knowledge_store/janitor.py` (see [`03-commit-write-path.md`](03-commit-write-path.md)).
|
||||
|
||||
## Canary findings (2026-07-30)
|
||||
|
||||
The first live agent turn on the canary workspace exposed two defects in the same write path, neither of which the suite caught.
|
||||
|
||||
**A delegated write never reached git.** The subagent wrote into copy `thread-21::task:call_x` while the end-of-turn commit, which only ever knows the parent thread, diffed `thread-21`. `diff_working_copy` raised `FileNotFoundError`, the commit returned `None` silently, and the copy leaked because `discard_working_copy` was never reached. Delegation is the normal path for agent writes, so most writes were affected. Fixed by scoping the copy to the turn's root thread (item 2). The tests missed it because all of them build the copy id by hand — `open_working_copy(f"thread-{THREAD_ID}")` — so none exercised the two sides *deriving* the id from a thread. The regression test now writes through `GitTreeBackend` with a `::task:` runtime and commits with the parent id.
|
||||
|
||||
**The legacy path silently covered for it.** `write_file` and `edit_file` set `dirty_paths` under `is_cloud` with no backend check, so `kb_persistence` recorded the same write into Postgres (`creates=1` in the turn log) and the document appeared in the UI. That masked the first defect — nothing errored — and fixing the copy id alone would have converted the silent loss into double writes: one revision plus a legacy document git never hears about. `dirty_paths` is one of five keys the legacy commit triggers on (with `staged_dirs`, `pending_moves`, `pending_deletes`, `pending_dir_deletes`); with all six mutating tools guarded, none is reachable under the git backend, so the legacy commit is a true no-op on a flipped workspace. `files` stays — it is the in-turn read cache, not a trigger.
|
||||
|
||||
Canary fallout was one Postgres-only document and one orphaned worktree. The worktree was discarded; the document needs no repair, since the seeder adopts it into git on the next flip.
|
||||
|
||||
## Tests
|
||||
|
||||
- ✅ Lifecycle: open at current revision / reopen in place / bare-dir on empty store / isolated parallel copies / net diff (adds, edits, deletes) / discard / age-based prune. (`tests/unit/knowledge_store/`)
|
||||
- ✅ Adapter: writes land on the turn's copy; tool calls share one copy; a subagent's write is visible to the orchestrator; committed content readable and deletable; mkdir→write; move; non-`/documents` paths rejected; per-thread isolation. (`tests/unit/middleware/test_git_tree_backend.py`)
|
||||
- ✅ A delegated write (namespaced `::task:` thread id) lands in the parent turn's revision, and its copy is discarded rather than orphaned. (`tests/integration/knowledge_store/test_commit_turn.py`)
|
||||
- ✅ `write_file`/`edit_file` stage nothing for the legacy commit under the git backend, while a workspace still on the old path keeps staging. (`tests/unit/middleware/test_git_tree_tool_staging.py`)
|
||||
- ✅ Resolver returns the git-tree adapter only when the flag is on; falls back to `KBPostgresBackend`/`StateBackend` otherwise.
|
||||
- ⏳ `read_file` envelope: raw line-numbered file, one `cite="[n]"` handle in the opening tag, no markers in file bytes; normalizer resolves `[n:Lx-Ly]` clamped to the document length at that revision.
|
||||
- ✅ `read_file` description matches what each mode returns: cloud-on-Postgres is told about the `view="full"` envelope, while git-native **and desktop** are told their reads carry no `[n]` and must not reuse a search ordinal. (`tests/unit/middleware/test_read_file_description.py`)
|
||||
|
||||
## Out of scope
|
||||
|
||||
- End-of-turn commit → Phase 3. Index refresh (vector-store-sync consumer) → Phase 4.
|
||||
- Desktop-local (`MultiRootLocalFolderBackend`) path — unchanged.
|
||||
- Other adapters (KB REST API, MCP) — deferred (ADR 0002, YAGNI).
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
# Phase 3 — Commit-per-turn write path
|
||||
|
||||
> Build after Phase 2 (needs the working-copy backend) and Phase 1's `transaction`. Umbrella: [`00-umbrella-plan.md`](00-umbrella-plan.md).
|
||||
>
|
||||
> **Status: SHIPPED (2026-07-29).** All six work items landed; see per-item notes for the small deviations from the locked model (receipt derivation, message model seam).
|
||||
|
||||
## Objective
|
||||
|
||||
Turn the agent turn's working copy — plus editor saves and upload-extracted markdown — into **one atomic revision** per turn/save. This is where the single write path for all indexed content lands.
|
||||
|
||||
## Locked model
|
||||
|
||||
- **One revision per agent turn.** A new, small persistence middleware (alongside `kb_persistence`, same package area): `aafter_agent` → `diff_working_copy(thread-{id})` → one `KnowledgeStore.transaction` scope replaying the net writes/removes → `discard_working_copy`. Empty diff records nothing. The commit body is a **free function** so the stream-task disconnect fallback (`event_loop.py`) can run the identical routine when `aafter_agent` is skipped — same shape as today's `commit_staged_filesystem_state`.
|
||||
- **`kb_persistence` is not touched.** Both middlewares coexist behind `KNOWLEDGE_STORE_ENABLED`; the old path keeps serving unflagged workspaces. Deletion of `kb_persistence`, `KBPostgresBackend`, `revert_service`, and the revision models happens at cut time (Phase 5), once every workspace is migrated and verified.
|
||||
- **Aider-style commit messages.** A model generates a one-line Conventional Commits subject from the turn's diff; `Thread: {id}` trailer carries provenance. *As built:* no weak/fast model role exists yet (`LLMRole` has only `AGENT`), so the generator takes any chat model as its seam and is wired with the workspace agent LLM — a dedicated weak model is a one-argument swap. Generation failure (or the LLM-less disconnect path) falls back to a deterministic subject; a commit is never lost to message generation.
|
||||
- **Honest attribution (Aider split).** Author = the acting user; committer = the agent identity. Autonomous writes (no user) author as the agent. `record`/`transaction` gained a `committer` parameter (defaults to author); conventions live in `knowledge_store/identities.py`.
|
||||
- **Receipts survive, derived from the recorded diff.** *As built:* the middleware **creates** receipts post-commit from `list_changes(revision)` (same ground-truth discipline as the old commit body — no provisional flip needed on this path), revision id as `external_id`. On commit failure it returns `failed` receipts and **keeps the copy** so the thread's next turn recovers the work. Receipts are file-only: history tracks content, so a directory's existence is proven by the receipt of the first file written into it.
|
||||
- **No Zero events here.** `document_created/updated/deleted` dispatches move with the derived rows (Phase 4) and projection (Phase 6); flag-on workspaces are dev/test until then.
|
||||
- **Janitor.** Celery beat task (daily, 4:45) sweeps every workspace via `knowledge_store/janitor.py` → `prune_working_copies(older_than_seconds=24h)`. 24h far exceeds any turn; crashed-turn copies are reused (and committed) by the thread's next turn well before that.
|
||||
- **Editor saves & upload-extracted markdown use the same commit path** (one write path): `services/document_revision_recorder.py` resolves the document's canonical path with the existing `doc_to_virtual_path` resolver (the same `/documents/...` namespace agents see) and records one `transaction` per save, behind the flag. Editor messages are deterministic (`docs: save <filename>`) — there is no chat context to summarize. A retitle drops the document's previous file in the same revision (the last recorded path is remembered on the row's `PATH_MARKER` metadata), or one document becomes two files. During coexistence a recording failure logs instead of failing the already-committed Postgres save; that flips at the Phase 5 cut.
|
||||
- **Connector-indexable sync records at the pipeline choke point**: every indexer (Notion, Drive, Confluence, uploads, …) converges on `IndexingPipelineService.prepare_for_indexing`, so the recorder hooks there — right after the batch's markdown commits to Postgres — as **one revision per sync batch** (`sync: index N document(s)`), not one per document. Recording at content-durability time (not after chunking) means embedding/LLM failures can never block the record: git is truth, chunks are derived. This superseded the earlier per-upload call in `UploadDocumentAdapter.index`, which was deleted. Identical re-synced content is a natural no-op (unchanged tree → no revision).
|
||||
- **No Postgres content writes here** — chunk/embedding refresh is Phase 4 (triggered off the revision).
|
||||
|
||||
## Work items
|
||||
|
||||
1. ✅ New persistence middleware `main_agent/middleware/knowledge_store_persistence/` (`commit_turn.py` free-function body, `commit_message.py`, `middleware.py`, `builder.py`); wired into `stack.py` alongside `kb_persistence`, gated by cloud mode + `KNOWLEDGE_STORE_ENABLED`.
|
||||
2. ✅ `committer` parameter on `record`/`transaction`; `Revision` carries both identities.
|
||||
3. ✅ `event_loop.py` gained a second safety-net block calling the same free function — no state markers needed (the working copy on disk *is* the pending state; no copy = no-op, naturally idempotent).
|
||||
4. ✅ Revision id surfaced as every success receipt's `external_id` (reaches state via the existing receipts channel). A dedicated event for the indexer/projector lands with Phase 4's trigger wiring.
|
||||
5. ✅ Editor save (`editor_routes.save_document`) calls `record_saved_document`; uploads and all connector indexers are covered by `record_prepared_documents` inside `prepare_for_indexing` (one revision per sync batch).
|
||||
6. ✅ Celery beat janitor (`prune_knowledge_store_working_copies`, daily).
|
||||
|
||||
## Tests (shipped)
|
||||
|
||||
- ✅ One revision per turn with the turn's **net** changes; message carries subject + `Thread:` trailer; author = user, committer = agent. (`tests/integration/knowledge_store/test_commit_turn.py`, real git + real Redis)
|
||||
- ✅ A turn that never touches the KB records nothing; an untouched copy records nothing.
|
||||
- ✅ Lock contention yields `failed` receipts, keeps the copy, and the next commit recovers the work.
|
||||
- ✅ Message generation: model subject used; deterministic fallback on model failure and on the LLM-less path. (`tests/unit/middleware/test_commit_message.py`)
|
||||
- ✅ Builder gating: flag on + cloud only. Editor-save recording: one revision, author identity, filename in message. (`test_document_recorder.py`)
|
||||
- ✅ Janitor prunes only copies older than the TTL, across workspaces. (`tests/unit/knowledge_store/test_janitor.py`)
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Building the chunk/embedding index from the revision → Phase 4.
|
||||
- Zero row projection → Phase 6.
|
||||
- Deleting the legacy write path → Phase 5 (cut time).
|
||||
|
||||
## Resolved
|
||||
|
||||
- Commit message format → Aider-style, weak-model generated, `Thread:` trailer (was open question 1).
|
||||
- Squash policy → one revision per turn is the model; the working copy nets intra-turn noise by construction (was open question 2).
|
||||
- Author identity → Aider split (author = user, committer = agent).
|
||||
|
|
@ -1,91 +0,0 @@
|
|||
# Phase 4 — Derived index + reindex
|
||||
|
||||
> **Shipped.** `app/knowledge_store/index/` (`converge.py` + `queue.py`) plus the Celery wiring under `app/tasks/celery_tasks/knowledge_store/`; work items 1, 2, 4, 5, 6 are all in. Where the build differs from what was planned here, the plan text says so inline under **Built as**; those notes are the record of what changed and why. Umbrella: [`00-umbrella-plan.md`](00-umbrella-plan.md).
|
||||
>
|
||||
> **Unblocked.** Phase 1's `list_changes`/`list_paths`/`read_as_of` and Phase 3's write path both shipped. Nothing here depends on C2 — the dependency runs the other way: C2's excerpt render consumes work item 2's line spans, and its own four pieces (envelope, registry, normalizer, resolver) stay out of scope here.
|
||||
>
|
||||
> **Why this matters now:** for flagged workspaces `kb_persistence` no-ops (`middleware/stack.py:250-252` leaves the state overlay empty), so agent notes are committed to git and **have no document rows, chunks, or embeddings at all** — invisible to search. This phase closes that gap.
|
||||
|
||||
## Objective
|
||||
|
||||
Make Postgres a **derived, rebuildable** chunk/embedding index of the store: incremental on each revision (keyed by content id), fully reproducible via one `reindex(workspace)`. Git history replaces the three hand-rolled versioning systems (their deletion lands at the Phase 5 cut).
|
||||
|
||||
## Locked model
|
||||
|
||||
- **Post-revision incremental index.** On each revision, `list_changes(revision)` names the changed paths; re-chunk + re-embed only those. **Key embeddings by content id**: unchanged content → same id → reuse existing vectors (no re-embed). This is the correct, native form of what `indexing_pipeline/chunk_reconciler.py::reconcile` already approximates (it matches by chunk *text*; the content id generalizes it to file identity).
|
||||
- **Identity is the path, and it already lines up.** `compute_identifier_hash` (`indexing_pipeline/document_hashing.py`) and `generate_unique_identifier_hash` (`utils/document_converters.py`) build the same `{type}:{unique_id}:{workspace_id}` string, so a synthetic `ConnectorDocument(document_type=NOTE, unique_id=<virtual path>)` yields the identity the legacy path and `virtual_path_to_doc` already use. This is why Phase 4 is a **thin adapter over the existing pipeline**, not a second pipeline.
|
||||
- **One convergence function, two callers.** `index_revision` and `reindex` (shipped as `index_changes` / `index_tree`) differ only in which paths they pass and which rows they prune — they share the body. Determinism between the incremental and rebuild paths is then structural, not something a test hopes for.
|
||||
- **Document rows converge; they are never wiped.** `documents` and `folders` are in the Zero publication (`alembic/versions/116_create_zero_publication.py`), so their ids reach the browser. A rebuild upserts by `unique_identifier_hash` and deletes only rows whose path left the tree; wiping and recreating would make every note vanish and reappear with new ids. **Chunk rows are the disposable layer**, replaced per document by the existing pipeline.
|
||||
- **Everything stored must be derivable from git.** Anything threaded in from a caller (notably `created_by_id`) is erased by the next `reindex`, making the two paths disagree. Derive the actor from the revision author instead — `knowledge_store/identities.py::user_identity` encodes it as `<id>@users.surfsense`.
|
||||
- **Postgres is disposable.** A single idempotent **`reindex(workspace)`** rebuilds the index from the current revision (the Fossil `rebuild` discipline). Search (`shared/retrieval/hybrid_search.py`) is unchanged — it reads the same `chunks` table.
|
||||
- **History = git; deletion at cut time.** `utils/document_versioning.py` (`DocumentVersion`), `services/revert_service.py` + `DocumentRevision`/`FolderRevision` become dead code for flagged workspaces here, but stay running for unflagged ones — proven by a test, not by inspection (see Tests). The delete sweep (code + Alembic table drops) is Phase 5, after migration + verification.
|
||||
|
||||
## Work items
|
||||
|
||||
1. `app/knowledge_store/index/converge.py` — `index_revision(workspace_id, revision)` and `reindex(workspace_id)` over one shared `_converge(...)`. Paths come from `list_changes` / `list_paths` (shipped: added/modified/removed + content ids), content from `read_as_of`; each document is upserted then handed to `IndexingPipelineService.index()` wrapped in a synthetic `ConnectorDocument`. Removed paths drop their document row (chunks cascade).
|
||||
**Bypass `prepare_for_indexing`.** It silently drops a *new* path whose content matches an existing document and marks an *edited* one `failed("Duplicate content")` (`indexing_pipeline_service.py:279-311`). `cp a.md b.md` is legal in git and must yield two indexed documents — path is identity, content is not unique. Model the upsert on `kb_persistence/middleware.py::_create_document` and reuse its `ensure_folder_hierarchy` for folder rows.
|
||||
|
||||
**Built as** `index_changes(session, workspace_id)` and `index_tree(session, workspace_id)` — no revision parameter, and the session is the caller's. Four deviations, all forced by the code:
|
||||
- **Named for scope, not for destructiveness** (renamed 2026-07-30). `index_revision` promised an argument it does not take, and `reindex` is the same verb the *legacy* per-document editor path uses (`document_reindex_tasks.py`) — the writer this phase guards against. "Rebuild" would also have been a lie: the run upserts and prunes, and document ids survive it (see the "never wiped" note above). The two entry points differ only in scope, so the names say scope.
|
||||
- **No revision argument.** Two saves in a row enqueue two tasks and the index lock serializes them without ordering them, so stamping the older id last would leave a stale index. Both entry points converge to `get_current_revision()` under the lock, which makes task order irrelevant. The incremental plan folds *every* revision between the stamp and head, so a dropped task costs nothing.
|
||||
- **Rows are adopted by path, not just by NOTE hash.** An upload is already in the tree with a `FILE:<filename>` identity (`file_upload_adapter.py` writes the row, then records the same markdown to git), so a hash-only lookup inserts a second row and the file appears twice in search. Resolution is: ownership marker, then NOTE hash, then `virtual_path_to_doc`. Every touched row gets `document_metadata["virtual_path"]`, which is also the prune key. `ponytail:` an adopted row keeps its `FILE:` hash while its location is path-derived, so two identity formulas coexist; the marker bridges them and Phase 5's migration has to unify them anyway.
|
||||
- **Prune is keyed on that marker, never on the workspace or on `document_type`.** Slack, Notion and the folder indexers write rows in the same workspace with no path in the tree, so a workspace-minus-tree prune deletes all of them on the first rebuild; a `NOTE`-only prune leaks the other way and never deletes an adopted upload's row. A **partial failure withholds the stamp** so the sweep re-drives it, while an intentional skip (undecodable or blank blob) still stamps — otherwise one bad blob wedges the workspace into rebuilding forever.
|
||||
2. **Store each chunk's `start_line`/`end_line` at cut time.** Sole consumer: rendering true document line numbers on search excerpts ([`00c-shared-contract.md`](00c-shared-contract.md) C2) — never a stored reference the frontend follows, so rebuilds can't strand it. Bigger than it looks, and the **only** piece C2 needs from this phase:
|
||||
- Alembic migration adding both columns to `chunks` (neither exists today).
|
||||
- `chunk_text` discards chonkie's `start_index`/`end_index` (`document_chunker.py:19`), and `chunk_text_hybrid`'s `.strip()` destroys the offset mapping — absolute offsets need `segment_start + stripped_prefix + chunk.start_index`.
|
||||
- **Spans live in the cached value**, not recomputed downstream: they are a pure function of the cache's existing key (`markdown_sha256 + chunker_kind + chunker_version`), so a `chunker_version` bump is the whole invalidation story. Recovering offsets later by searching the source for chunk text is ambiguous whenever a document repeats a line (boilerplate, table rows).
|
||||
|
||||
**Built as** `attach_line_spans(text, chunks)` in `document_chunker.py`, called on both sides of the cache boundary rather than stored in the cached value, and **neither chunker's signature changed**:
|
||||
- The ambiguity the bullet above warns about only exists for a *whole-document* search. Chunks arrive in document order and do not overlap, so a left-to-right cursor resolves repeated text unambiguously — covered by a test for a document that repeats a line, and one whose table rows repeat.
|
||||
- Deriving spans at read time means **cached entries stay valid and the chunker version is not bumped**: no fleet-wide re-embed on merge, which is the expensive half of the planned approach. It also leaves `chunk_text`/`chunk_text_hybrid` and every test seam that patches them untouched, and keeps the hybrid chunker's `.strip()` from needing offset bookkeeping.
|
||||
- `ponytail:` ceiling — a chunker that emits overlapping windows or rewrites chunk text degrades to the cursor's line rather than failing; upgrade path is to thread chonkie's own `start_index` through, which is exactly the work this defers.
|
||||
- The reconciler had to change too: unchanged text still **moves** when a paragraph is inserted above it, so `reused` carries the new span and the position-only `UPDATE` became position-plus-span. Without that, editing a document leaves stale line numbers on every chunk below the edit.
|
||||
3. ~~Add a blob-SHA reuse layer~~ — **already shipped.** `indexing_pipeline/cache/cached_indexing.py::build_chunk_embeddings` caches the summary vector and every chunk vector under `EmbeddingKey(markdown_sha256, embedding_model, embedding_dim, chunker_kind, chunker_version)`: content-addressed, no workspace salt, i.e. the content id this phase wanted, and `index()` already routes through it. The legacy note path (`kb_persistence/middleware.py:235-239`) calls `chunk_text`/`embed_texts` directly and bypasses the cache, which is likely why it was believed missing. C5 still holds: **`content_hash` is workspace-salted, so it is NOT a content id** — do not alias them.
|
||||
4. `reindex(workspace_id)` behind a Celery task (mirror `knowledge_store/janitor_task.py`; register in `app/celery_app.py`). Serialize per workspace with an **index lock distinct from the write lock**: the write lock's 30s TTL is sized for a commit, and reusing it would stall agent writes behind embedding calls.
|
||||
|
||||
**Built as** `workspace_index_lock` beside `workspace_write_lock` over a shared `_workspace_lock(purpose, ttl, wait)`; the write lock keeps its key and its 30s TTL. A contender gives up in 5s rather than queueing — but what happens next depends on who lost (2026-07-30): a losing **rebuild** skips outright (the competing rebuild converges the same tree), while a losing **per-save task retries** on a 30s countdown (bounded at 10) — its save may have landed after the holder read HEAD, and without the retry a save arriving mid-converge would go stale until the hourly sweep. A redundant retry no-ops against the stamp for one HEAD read. Whole-workspace rebuilds route to the connectors queue; the per-save task stays on the fast queue since search freshness is user-facing.
|
||||
5. Trigger `index_revision` off Phase 3's surfaced revision id, at **both** writers (`commit_turn.py`, `services/document_revision_recorder.py`). Enqueue-only, never raising — the content is already committed either way.
|
||||
6. **Self-healing drift, not a deploy step.** A new `workspaces.last_indexed_revision` makes `last_indexed_revision != get_current_revision()` a drift predicate; a daily Beat task enqueues `reindex` for drifted flagged workspaces. That one mechanism covers the initial backfill of already-flagged workspaces (they have git content and no index today), a lost Celery task, a crashed worker, and any workspace flagged later. Runbook steps get forgotten; converging systems don't.
|
||||
|
||||
**Built as** `reindex_drifted_workspaces`, **hourly with a per-run enqueue cap**, not daily: hourly cuts recovery from a lost task from a day to an hour, and the cap bounds the fan-out, not the check — the drift check is one HEAD read per workspace, while each task it enqueues embeds. Candidates are **flipped workspaces only** (`workspaces.knowledge_store_enabled`, the Phase-5 per-workspace flag): a seeded-but-unflipped workspace has a repo too, but Postgres is still its write model, and indexing it would fight the legacy pipeline. A never-indexed candidate (stamp `NULL`) routes to the rebuild task on the connectors queue so a backfill can't bury user-facing saves; a stamped one takes the incremental task on the fast queue. The worker re-checks the flag before converging, because a queued task can outlive an unflip.
|
||||
|
||||
**This sweep only covers half the drift (2026-07-30).** Its predicate compares a stored git revision against the store's HEAD — both sides come from git, so it is structurally blind to drift on the Postgres side (a row git never received, content that disagrees, an orphan the change log never reported). Phase 5's `check_knowledge_store_drift` is what sees that half, comparing the two stores by content address, and it now **enqueues `index_tree` on a `drift` verdict** rather than logging for someone to act on — this item's own reasoning applied to the case it doesn't reach. Details and the capped fan-out: `05-migration.md` item 7.
|
||||
|
||||
## Tests
|
||||
|
||||
- **Identical content at two paths yields two documents** — the case `prepare_for_indexing` gets wrong, and the reason the upsert is hand-written.
|
||||
- Edit one file → only its chunks re-embed; an untouched file's chunk rows keep the same ids and byte-identical vectors (content-id reuse verified).
|
||||
- Each chunk's `start_line`/`end_line` matches the exact slice of the blob it was cut from — including a document whose text repeats, and one containing a Markdown table (the hybrid chunker's strip path).
|
||||
- `reindex(workspace)` produces a chunk set identical to the incremental path (determinism), and leaves document ids unchanged.
|
||||
- Re-running `index_revision` on an already-stamped revision is a no-op.
|
||||
- Deleting a file removes its document and chunks; renaming preserves vectors (same content id → cache hit, no model call).
|
||||
- **Search parity is differential, not a golden baseline.** Index identical content through the connector pipeline and through the git indexer; assert a fixed query returns the same documents in the same order. A stored baseline rots the first time the chunker or embedding model changes, and then someone deletes the test.
|
||||
- **A save in a flagged workspace creates zero `DocumentVersion` / `DocumentRevision` rows** — turns the locked model's dead-code claim into an enforced invariant before Phase 5 drops the tables under it. `routes/documents_routes.py` also writes versions; confirm whether that path is flag-gated.
|
||||
- Unit: a synthetic `ConnectorDocument`'s hash equals `generate_unique_identifier_hash(NOTE, virtual_path, workspace_id)`. The whole adapter rests on two formulas in separate modules agreeing.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Live connectors (Slack/Gmail) — never indexed.
|
||||
- Zero row *projection* (folder/document rows driven from the tree as a first-class concern) → Phase 6. Note this is not the same as the existing publication: `documents`/`folders` already replicate, which is why rebuild stability is in the locked model rather than deferred.
|
||||
- C2's envelope, citation registry, `[n:Lx-Ly]` normalizer, and `read_as_of` resolution — Phase 2's remaining work item, unblocked by this phase's line spans.
|
||||
- Reranker/chunking strategy changes (separate search work).
|
||||
|
||||
## Resolved (see [`00c-shared-contract.md`](00c-shared-contract.md))
|
||||
|
||||
- **content_hash vs content id:** `content_hash` is workspace-salted, not a content id; reuse keys on the cache's unsalted `markdown_sha256`. Keep `content_hash` through migration, drop later if redundant (C5).
|
||||
- **Cache location:** `indexing_pipeline/cache/` — exists since 2026-06-12; extend it, don't build a new layer (C5, corrected).
|
||||
- **Where `reindex` runs:** Celery task (C5).
|
||||
- **Rebuild granularity:** document rows converge (upsert + prune); only chunk rows are wiped. `documents`/`folders` are in the Zero publication, so their ids reach the browser and must be stable across a rebuild.
|
||||
- **Rename semantics:** a rename lands as removed + added, so the document gets a new path-derived identity and a fresh row; vectors survive via the content-addressed cache. Accepted — path is identity, and rows are derived data.
|
||||
|
||||
## Also shipped (not planned here)
|
||||
|
||||
Both are single writers colliding on one document's chunk rows once a workspace is git-backed. Neither is Phase 4 work by the letter of this plan; both had to land with it or the phase ships a corruption.
|
||||
|
||||
- **`reindex_document_task` no-ops for flagged workspaces**, guarded inside `_reindex_document` rather than at its two call sites (`editor_routes.py`, `documents_routes.py`) so neither can be missed. It re-chunks from Postgres `source_markdown` and titles the document from its first heading, while the indexer re-chunks from git and titles it from the filename stem — both running flips the title on every save.
|
||||
- **`restore_document_version` returns 409 for flagged workspaces.** It rewrites `source_markdown` and `title` without recording a revision, so search would keep serving git's newer content and the next `reindex` would revert the restore outright. History is `git revert` for these workspaces, and the route is deleted at the Phase 5 cut. This is also the only writer of `DocumentVersion` rows on the save path, which is what makes this plan's "zero version rows" test an invariant rather than an intention.
|
||||
- Adjacent fix: all three `/documents/{id}/versions*` routes passed a `User` where `check_permission` expects an `AuthContext`, so every one of them 500'd. Corrected while adding the 409, since the test could not otherwise reach the guard.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. ~~`reindex` progress/observability surface (log vs. status row)~~ — **decided:** one structured log line per converge run (revision, indexed, skipped, failed, deleted, stamped). Withholding the stamp on failure makes the failure case self-healing, which is what a status row would have been read for.
|
||||
|
|
@ -1,152 +0,0 @@
|
|||
# Phase 5 — Migration
|
||||
|
||||
> **Status: TOOLING SHIPPED (2026-07-30); fleet flips pending.** Seeder (`app/knowledge_store/migrate.py`), fleet runner (`scripts/migrate_knowledge_store.py`), per-workspace flag (`workspaces.knowledge_store_enabled`), drift monitor. No production workspace flipped yet; cut-time deletion (versioning code + table drops) runs after fleet verification.
|
||||
>
|
||||
> After Phases 1–4. One-time, per-workspace, behind the flag. Umbrella: [`00-umbrella-plan.md`](00-umbrella-plan.md).
|
||||
>
|
||||
> **Executing it:** the ordered commands and checks for a production run live in [`05a-seed-runbook.md`](05a-seed-runbook.md) — merge, deploy, pre-flight, dry run, seed, verify, flip in batches, watch, roll back.
|
||||
|
||||
## Objective
|
||||
|
||||
Move each existing workspace's KB from Postgres-as-truth to git-as-truth by exporting current documents/folders into an initial git repo, then flipping `KNOWLEDGE_STORE_ENABLED` for that workspace once content identity is verified. **Adopt the existing derived index; never rebuild it during migration.**
|
||||
|
||||
## Why "adopt, don't rebuild" (amended 2026-07-29)
|
||||
|
||||
The first draft gated the flip on `seed → reindex() → compare search results`. A full
|
||||
`reindex` re-chunks and re-embeds every document — the same cost class as the chunk-table
|
||||
backfill we once measured at ~21 days and abandoned. It is also **unnecessary**: the seed
|
||||
copies bytes *out of Postgres*, so the existing `chunks` rows and their vectors are already
|
||||
the correct derived index of the seeded repo. Verifying bytes proves the index; rebuilding
|
||||
it proves nothing extra and costs weeks plus embedding spend.
|
||||
|
||||
This is the standard online-migration shape (Stripe's dual-write → backfill → verify →
|
||||
cutover → delete; expand/contract): Phase 3's flag-gated dual-run **is** the dual-write
|
||||
step, the seed **is** the backfill, byte parity **is** the shadow-read verification, the
|
||||
flag flip **is** the cutover, and the Phase-5 delete sweep **is** the contract.
|
||||
Re-embedding is never on that path.
|
||||
|
||||
## Locked model
|
||||
|
||||
- **Seed commit per workspace.** Read current `folders` + `documents`
|
||||
(`source_markdown`/`content`) and write the real tree into the Phase-1 repo as **one
|
||||
seed commit** (`author=migration`), using the same path rules as the live write path
|
||||
(C1). Streamed table scan + file writes: O(documents) I/O, no embeddings, no locks on
|
||||
hot tables. Idempotent — re-seeding unchanged content is a no-op commit.
|
||||
- **Preserve identity.** Keep the `unique_identifier_hash` ↔ path mapping so connector
|
||||
re-syncs and existing references stay stable.
|
||||
- **Parity = byte identity, not reindex.** Gate the flip on: every seeded blob's bytes
|
||||
equal the document's Postgres markdown (and nothing is missing/extra). O(documents)
|
||||
hashing, seconds per workspace. `reindex()` stays a disaster-recovery tool; run it once
|
||||
on one small pilot workspace as a one-time Phase-4 sanity check, never as a
|
||||
per-workspace gate.
|
||||
- **The seed revision is adopted, never incrementally indexed** (contract C7). To
|
||||
Phase 4's `index_revision`, the seed looks like "every file added" — feeding it through
|
||||
would re-embed the whole workspace (the storm the parity redesign exists to avoid). The
|
||||
seeder marks the seed revision as the indexer's starting point; incremental indexing
|
||||
begins with the first post-seed revision.
|
||||
- **No span work in migration (amended 2026-07-29, second pass).** New chunks get
|
||||
`start_line`/`end_line` at cut time (nullable columns, instant ALTER — C2); legacy
|
||||
chunks stay `NULL`, render un-numbered, and cite at document level (fail-closed
|
||||
normalizer). Convergence is a **separate deadline-free daily fill job** — CPU-only,
|
||||
matching stored chunk texts against the git blob (post-flip truth; all-or-nothing per
|
||||
document, see C2), most-used workspaces first — that is operational work, never a
|
||||
migration step or flip gate. Migration
|
||||
itself touches the chunks table zero times; the mandatory-backfill mistake (PR #1523,
|
||||
the ~21-day job) stays dead.
|
||||
- **Chunker drift converges lazily, on edit.** Migrated chunks were cut by whatever
|
||||
chunker was live at index time. Do not re-chunk them eagerly and do **not** warm the
|
||||
embedding cache from legacy rows (entries would be keyed under the current
|
||||
`chunker_version` for boundaries it did not produce — cache poisoning). On a document's
|
||||
next edit, the normal pipeline re-chunks it under the current version; the reconciler
|
||||
and the embedding cache bound the cost to what actually changed.
|
||||
- **Rollback window.** Keep Postgres content intact until the flagged workspace is
|
||||
verified; flag flip is the point of no return per workspace.
|
||||
|
||||
## Work items
|
||||
|
||||
1. ✅ `app/knowledge_store/migrate.py` (2026-07-29) — `migrate_workspace(session, workspace_id)`:
|
||||
builds the tree from `documents` via the live path rules (`build_path_index` +
|
||||
`doc_to_virtual_path`, identical paths to every write path), falls back to `content`
|
||||
for rows predating `source_markdown`, one seed revision (`author=MIGRATION_IDENTITY`).
|
||||
The DB-free core `seed_workspace(workspace_id, files, dry_run=)` carries the tests.
|
||||
Seed = "make the tree exactly this": a catch-up re-seed also **removes** documents
|
||||
deleted in Postgres since the prior seed, so seed→(activity)→re-seed→flip converges.
|
||||
Unlike the recorder it does **not** guard on `KNOWLEDGE_STORE_ENABLED` — migration
|
||||
runs before the flip by definition.
|
||||
2. ✅ Parity check (same run): content-address comparison via `list_paths` +
|
||||
`compute_content_id` — zero file reads — reporting `missing`/`extra`/`mismatched`;
|
||||
`MigrationReport.ok` is the flip guard's verdict. Report, don't fix.
|
||||
3. ⏳ Seed adoption: the report surfaces `seeded_revision`; recording it as the
|
||||
indexer's last-indexed point is Phase 4's side of C7 (coordinate the bookkeeping
|
||||
shape with `index_revision`).
|
||||
4. ✅ Per-workspace flag flip guarded by `report.ok` (2026-07-29) —
|
||||
`workspaces.knowledge_store_enabled` (migration 175, default false) AND the global
|
||||
`KNOWLEDGE_STORE_ENABLED` (kept as the master kill switch: env off = everything off,
|
||||
instantly). `knowledge_store_enabled_for(workspace_id)` resolves the pair (30s
|
||||
per-process cache on the workspace half). The agent factory resolves it **once per
|
||||
turn** and passes the verdict down (resolver, persistence middleware, compiled-graph
|
||||
cache key — the flag rotates cached graphs), so a turn never mixes write paths; the
|
||||
recorder and the disconnect safety-net check per call. The fleet runner flips:
|
||||
`--yes --flip` (only ever on a passing report), `--unflip --workspace N` rolls back.
|
||||
Flipping back loses nothing — Postgres is updated in both modes; git goes stale and
|
||||
a catch-up re-seed converges it.
|
||||
5. ✅ Dry-run mode: skips the write, reports parity against head, creates nothing for
|
||||
fresh workspaces.
|
||||
6. ✅ Fleet runner `scripts/migrate_knowledge_store.py`: dry-run by default, fresh
|
||||
session per workspace, append-only JSONL reports, non-zero exit on any not-ok.
|
||||
7. ✅ Drift instrumentation (2026-07-29) — the coexistence window is watched, not
|
||||
trusted. Every recording attempt emits
|
||||
`surfsense.knowledge_store.record.outcome` (`flow` = editor_save / sync_batch /
|
||||
turn_commit; `status` = recorded / noop / failed): a non-zero `failed` rate means
|
||||
git is falling behind Postgres. A daily beat task (`check_knowledge_store_drift`,
|
||||
05:15) runs the seeder's dry-run parity over every **flipped** workspace and emits
|
||||
`surfsense.knowledge_store.drift.check` (ok / drift / error) per workspace, with a
|
||||
warning log naming the missing/extra/mismatched paths — the JSONL report as an
|
||||
always-on alarm instead of a by-hand check.
|
||||
|
||||
**Amended 2026-07-30 — the monitor repairs, it does not just alarm.** Phase 4's
|
||||
hourly sweep compares a stored git revision against the store's HEAD, so *both*
|
||||
sides of its predicate come from git: it is structurally blind to drift that
|
||||
lives on the Postgres side, which is exactly what this check sees. Leaving that
|
||||
half to `reindex_knowledge_store.delay(...)` typed by hand contradicted Phase 4's
|
||||
own "runbook steps get forgotten; converging systems don't" — same class of
|
||||
fault, two different answers. A `drift` verdict now enqueues the whole-tree
|
||||
converge (`index_tree` upserts paths Postgres lacks, overwrites content that
|
||||
disagrees, prunes marked rows whose file is gone), capped at
|
||||
`REPAIR_ENQUEUE_CAP = 10` per run: fleet-wide drift is a systemic fault, and
|
||||
fanning out rebuilds would compound it. `error` stays alarm-only — a store the
|
||||
check could not read is not fixed by indexing it harder, and a failed report's
|
||||
parity fields describe nothing. Known ceiling, marked in the code: drift
|
||||
`index_tree` cannot fix (an unmarked Postgres row with no file in the tree, i.e.
|
||||
a writer bypassing git) costs one rebuild per run until a human intervenes; the
|
||||
alarm persists throughout, and the upgrade path is a per-workspace attempt count.
|
||||
|
||||
## Tests
|
||||
|
||||
- ✅ Seed records one revision, passes parity, migration-authored.
|
||||
- ✅ Idempotent: re-seeding unchanged content records nothing.
|
||||
- ✅ Parity names missing/extra/mismatched paths; drift fails `ok`.
|
||||
- ✅ Dry-run builds nothing on fresh workspaces; passes parity on seeded ones.
|
||||
- ⏳ `unique_identifier_hash` mapping preserved (connector docs still resolve) — needs a
|
||||
Postgres-backed test or the pilot dry run.
|
||||
- ⏳ Adopted seed: `index_revision` on the first post-seed revision touches only that
|
||||
revision's changed paths (lands with Phase 4).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Binary re-import (blobs stay in the blob store).
|
||||
- Frontend cutover (separate umbrella).
|
||||
- Eager re-chunking of migrated content (lazy, on edit — see locked model).
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Big-bang all-workspaces vs. staged per-workspace rollout order (recommend staged).
|
||||
|
||||
## Sources
|
||||
|
||||
- Stripe, "Online migrations at scale" — dual-write → backfill → verify reads → cut
|
||||
writes → delete old (https://stripe.com/blog/online-migrations).
|
||||
- Expand/contract (parallel change) — old and new coexist through every step; never a
|
||||
breaking change in one step.
|
||||
- Lazy backfill discipline — fill only when absent; never rewrite an existing value
|
||||
because derivation logic changed; pair lazy convergence with a bounded background job.
|
||||
|
|
@ -1,454 +0,0 @@
|
|||
# Phase 5a — Production seed & flip runbook
|
||||
|
||||
> Operational companion to [`05-migration.md`](05-migration.md). That subplan says *what* the migration
|
||||
> is and why; this one is the ordered list of commands and checks for doing it on production.
|
||||
> Umbrella: [`00-umbrella-plan.md`](00-umbrella-plan.md).
|
||||
|
||||
**Read the safety property first.** Merging and deploying this work changes nothing at runtime,
|
||||
because git-native behaviour needs **both** flags on: the process-wide `KNOWLEDGE_STORE_ENABLED`
|
||||
(defaults `FALSE`, `app/config/__init__.py:543`) and the per-workspace
|
||||
`workspaces.knowledge_store_enabled` (defaults false, migration 175). Seeding writes only to git and
|
||||
to one metadata field; it never inserts or deletes a document row. So stages 0–6 are reversible by
|
||||
doing nothing, and stage 7 is the first one that changes how a workspace behaves.
|
||||
|
||||
Stage numbering is the execution order. Each stage lists **checks** (verify before moving on) and
|
||||
**stop conditions** (abort, do not continue).
|
||||
|
||||
---
|
||||
|
||||
## Stage 0 — Merge
|
||||
|
||||
The branch is `kb_git_mvp` on the fork (`origin` = `CREDO23/SurfSense`); upstream is
|
||||
`MODSetter/SurfSense`.
|
||||
|
||||
**One PR: `kb_git_mvp → upstream/main`.** The merge is conflict-free, so there is nothing to gain from
|
||||
routing through `dev` first. `main` already takes feature branches directly — `#1623`, `#1619`,
|
||||
`#1617` from this fork all landed that way — so this is the repo's normal path, not a shortcut.
|
||||
|
||||
Why not promote `dev → main` instead: that ships everything sitting in `dev`, and the two are not
|
||||
level. At the time of writing `dev` is **46 commits ahead** of `main` (searxng fallback,
|
||||
model-connection fixes, an automations fix). Promoting would put all of it into the same production
|
||||
deploy as this migration, giving two unrelated changes one blast radius.
|
||||
|
||||
The merge was **conflict-free** when last checked: this branch and `main` share the base `06c7e27c7`
|
||||
(2026-07-24), `main` has moved 33 commits since, and none of them touch our files. Re-verify before
|
||||
opening the PR — exit 0 means clean:
|
||||
|
||||
```bash
|
||||
git fetch upstream
|
||||
git merge-tree --write-tree upstream/main kb_git_mvp >/dev/null; echo "main: $?"
|
||||
```
|
||||
|
||||
### Two rules
|
||||
|
||||
**Merge, never squash or rebase.** The repo allows all three, but its practice is merge commits, and
|
||||
here it is load-bearing: our 127 commits must keep their SHAs on `main`, so that when someone later
|
||||
syncs `main` into `dev` git recognises them as common history. A squash rewrites them into one new
|
||||
SHA, and that sync becomes a re-application of the whole migration onto a branch that already has the
|
||||
same content — conflicts across every file we touched.
|
||||
|
||||
**Merge only `upstream/main` into the branch, never `upstream/dev`.** Merging `dev` in would drag its
|
||||
46 unreleased commits into the PR, which is the coupling this route exists to avoid.
|
||||
|
||||
### The PR
|
||||
|
||||
1. Merge `main` into the branch — and only `main`:
|
||||
```bash
|
||||
git fetch upstream && git merge upstream/main
|
||||
```
|
||||
2. Run what CI runs, and note the baseline:
|
||||
```bash
|
||||
cd surfsense_backend
|
||||
uv run pytest tests/unit tests/integration -q -p no:randomly --maxfail=100
|
||||
uv run ruff check . && uv run ruff format --check .
|
||||
```
|
||||
3. Open the PR against `upstream/main` and wait for `backend-tests`, `code-quality`, and `e2e-tests`.
|
||||
|
||||
**Checks**
|
||||
|
||||
- [ ] Alembic has a **single head** (`uv run alembic heads` → one revision, ours is `176`). Neither
|
||||
`main` nor `dev` has added a migration since our base, so this passes today; re-check after the
|
||||
merge anyway, because two heads makes the `migrations` container fail and halts the whole stack.
|
||||
- [ ] Migrations 175 and 176 are **additive only** (new nullable columns) — no row rewrite, no long
|
||||
lock.
|
||||
- [ ] `KNOWLEDGE_STORE_ENABLED` is **not** set true anywhere in the diff (compose, `.env.example`).
|
||||
- [ ] `VERSION` resolves to `main`'s value (`0.0.35`, not this branch's stale `0.0.34`) — it decides
|
||||
the image tag in stage 1.
|
||||
|
||||
**Follow-up this creates:** `dev` won't have these 127 commits, and `main` won't have dev's 46, so
|
||||
`dev` stops being a strict ancestor of `main`. Someone should merge `main` back into `dev` — the repo
|
||||
already does this after a release (`Merge commit 'a89b3aa2...' into dev`). Content-wise it is clean
|
||||
today: `git merge-tree --write-tree upstream/dev kb_git_mvp` also exits 0, so the two sets of changes
|
||||
don't overlap. Tell whoever owns `dev`, and do it before they open the next `dev → main` promotion.
|
||||
|
||||
**Stop if** the test baseline has failures beyond the known pre-existing ones. As of the merge:
|
||||
`6 failed, 3995 passed, 13 errors` — 3 in `automations`, 2 in `google_maps` parsers, 1 PAT static
|
||||
check, and the 13 errors are `google_maps` tests whose captured fixture JSON isn't in the repo
|
||||
(`FileNotFoundError` on `fixtures/boq_reviews_page.json`). None are in `knowledge_store`, and our diff
|
||||
touches none of those paths — confirm the same way: `git diff --name-only upstream/main...HEAD`.
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 — Deploy (still inert)
|
||||
|
||||
The backend image serves four roles from one build, dispatched by `SERVICE_ROLE`
|
||||
(`scripts/docker/entrypoint.sh:146-160`): `migrate` (one-shot, runs `alembic upgrade head` then
|
||||
exits 0), `api`, `worker`, `beat`.
|
||||
|
||||
### What the merge to `main` publishes
|
||||
|
||||
`main` is the default branch, so that push runs the full build chain
|
||||
(`.github/workflows/docker-build.yml`): it reads the `VERSION` file, finds the newest existing
|
||||
`X.Y.Z.N` tag, and increments the build number. The images are pushed as
|
||||
`ghcr.io/modsetter/surfsense-backend:X.Y.Z.<N+1>` **and** `:latest` (the `latest` alias is applied
|
||||
only for the default branch or a `v*` tag, line 341), then `finalize_release` pushes the git tag.
|
||||
With `VERSION` at `0.0.35` and build tags running to `0.0.35.2`, expect `0.0.35.3`.
|
||||
|
||||
This is also why the PR has to target `main` to be deployable at all: version computation is gated on
|
||||
the default branch, so a merge into `dev` builds images but nothing gets a version tag or the `latest`
|
||||
alias.
|
||||
|
||||
- [ ] Note the exact version tag the build produced, and deploy **that** rather than trusting
|
||||
`latest` to have settled.
|
||||
- [ ] All four backend services must move to the same tag together — a worker on an older image than
|
||||
the API is the same failure mode as a split volume. Note that the compose services carry
|
||||
`com.centurylinklabs.watchtower.enable=true` labels: if anything Watchtower-like is watching
|
||||
`latest` in your deployment, services can update unattended and at different moments. Pin the
|
||||
version tag for this deploy so the fleet moves as one.
|
||||
|
||||
Then five checks on the deployed stack. The first is the one that can corrupt data; the rest are the
|
||||
ones that fail quietly.
|
||||
|
||||
### Check 1.1 — the object-store volume is the *same* volume on api and worker
|
||||
|
||||
Git repositories live under `{FILE_STORAGE_LOCAL_PATH}/knowledge_store/{workspace_id}`
|
||||
(`app/config/__init__.py:546-549`), i.e. on the `object_store` volume mounted at
|
||||
`/app/.local_object_store` (`docker/docker-compose.yml:119`, `184`). The API writes editor saves and
|
||||
the worker runs indexing, so if these two mount **different** volumes, each sees its own repository:
|
||||
editor saves land in one, the index is built from the other, and the drift monitor will fight itself
|
||||
forever. Prove they share one:
|
||||
|
||||
```bash
|
||||
# in the api container
|
||||
echo "$(date -u +%FT%TZ) api" > /app/.local_object_store/_volume_probe
|
||||
# in the worker container
|
||||
cat /app/.local_object_store/_volume_probe # must print what api wrote
|
||||
rm /app/.local_object_store/_volume_probe
|
||||
```
|
||||
|
||||
- [ ] The worker reads what the API wrote.
|
||||
- [ ] `/shared_tmp` is likewise the same volume on both (already required for uploads,
|
||||
`Dockerfile:129-131`).
|
||||
- [ ] `beat` does **not** need either volume — it only schedules.
|
||||
|
||||
### Check 1.2 — the worker consumes the connectors queue
|
||||
|
||||
`reindex_knowledge_store` (the full-tree repair, and what the drift monitor auto-enqueues) is routed
|
||||
to `{default}.connectors` (`app/celery_app.py:268`). With `CELERY_QUEUES` unset, the entrypoint
|
||||
subscribes to default + `.connectors` + `.gateway` (`entrypoint.sh:95-104`). If someone has pinned
|
||||
`CELERY_QUEUES` to just `surfsense`, every repair silently queues forever.
|
||||
|
||||
```bash
|
||||
# in the worker container
|
||||
echo "CELERY_QUEUES=${CELERY_QUEUES:-<unset, good>}"
|
||||
celery -A app.celery_app inspect active_queues 2>/dev/null | grep -E "surfsense"
|
||||
```
|
||||
|
||||
- [ ] `CELERY_QUEUES` unset, or includes `surfsense.connectors`.
|
||||
|
||||
### Check 1.3 — beat is actually running
|
||||
|
||||
The hourly sweep, the janitor, and the drift monitor are beat entries
|
||||
(`app/celery_app.py:345-366`). Without beat there is no automatic recovery for a lost index task.
|
||||
|
||||
- [ ] A `SERVICE_ROLE=beat` service exists and its log shows the scheduler starting.
|
||||
|
||||
### Check 1.4 — schema is at head
|
||||
|
||||
```bash
|
||||
# in the api container
|
||||
alembic current # expect: 176 (head)
|
||||
```
|
||||
|
||||
- [ ] `176 (head)`, single head.
|
||||
|
||||
### Check 1.5 — disk
|
||||
|
||||
Git will hold a second copy of every document's markdown (working tree plus compressed objects).
|
||||
Estimate before you commit to it:
|
||||
|
||||
```sql
|
||||
SELECT pg_size_pretty(sum(length(coalesce(source_markdown, content)))::bigint) AS corpus,
|
||||
count(*) AS docs
|
||||
FROM documents
|
||||
WHERE coalesce(source_markdown, content) IS NOT NULL
|
||||
AND coalesce(source_markdown, content) <> 'Pending...';
|
||||
```
|
||||
|
||||
Budget roughly **2.5×** that figure on the `object_store` volume (working tree + objects + head-room
|
||||
for future revisions), on top of what the blob store already uses.
|
||||
|
||||
- [ ] Free space on the volume ≥ 2.5 × corpus, with margin.
|
||||
|
||||
**Stop if** any of 1.1–1.4 fails. None of them are recoverable by continuing.
|
||||
|
||||
---
|
||||
|
||||
## Stage 2 — Pre-flight state (nothing written yet)
|
||||
|
||||
Run against the production database (psql inside the `db` container, or your usual client):
|
||||
|
||||
```sql
|
||||
-- 2.1 the columns exist
|
||||
SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'workspaces'
|
||||
AND column_name IN ('knowledge_store_enabled', 'last_indexed_revision');
|
||||
|
||||
SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'chunks' AND column_name IN ('start_line', 'end_line');
|
||||
|
||||
-- 2.2 nothing is flipped yet
|
||||
SELECT count(*) FILTER (WHERE knowledge_store_enabled) AS flipped,
|
||||
count(*) AS total
|
||||
FROM workspaces;
|
||||
|
||||
-- 2.3 no stamps yet
|
||||
SELECT count(*) FROM workspaces WHERE last_indexed_revision IS NOT NULL;
|
||||
|
||||
-- 2.4 the shape of the job: documents per workspace
|
||||
SELECT workspace_id, count(*) AS docs,
|
||||
pg_size_pretty(sum(length(coalesce(source_markdown, content)))::bigint) AS bytes
|
||||
FROM documents
|
||||
WHERE coalesce(source_markdown, content) IS NOT NULL
|
||||
AND coalesce(source_markdown, content) <> 'Pending...'
|
||||
GROUP BY workspace_id ORDER BY 2 DESC;
|
||||
```
|
||||
|
||||
**Checks**
|
||||
|
||||
- [ ] 2.1 returns both workspace columns and both chunk columns.
|
||||
- [ ] 2.2 shows `flipped = 0`.
|
||||
- [ ] 2.3 shows `0`.
|
||||
- [ ] 2.4 gives you the batch plan: note the biggest workspaces and pick 2–3 **small, internal** ones
|
||||
as the canary set.
|
||||
- [ ] `KNOWLEDGE_STORE_ENABLED` is still off in the api/worker environment (`env | grep KNOWLEDGE`).
|
||||
|
||||
---
|
||||
|
||||
## Stage 3 — Seed dry run (writes nothing)
|
||||
|
||||
Run **inside a container that mounts the object store** — the api or worker, not a fresh one-off
|
||||
container, or the seeder will inspect an empty volume and report the whole fleet as missing.
|
||||
|
||||
```bash
|
||||
mkdir -p /app/.local_object_store/ks-migration
|
||||
cd /app && python scripts/migrate_knowledge_store.py \
|
||||
--out /app/.local_object_store/ks-migration/dry-$(date -u +%Y%m%dT%H%M%SZ).jsonl
|
||||
```
|
||||
|
||||
Writing the report onto the volume matters: the container filesystem is ephemeral, and this file is
|
||||
your audit trail and your resume point.
|
||||
|
||||
**Expected output.** Every unseeded workspace reads
|
||||
`drift: missing=N extra=0 mismatched=0, N file(s)`, and the command **exits 1**. That is normal
|
||||
pre-seed — a workspace that has not been seeded is not "ok". What matters is the prefix.
|
||||
|
||||
**Checks**
|
||||
|
||||
- [ ] No line begins `error:` — that is a real failure (unreadable store, mapping bug), not "not
|
||||
seeded yet". Triage before seeding.
|
||||
- [ ] `extra=0` and `mismatched=0` everywhere. Non-zero here on a *fresh* store means something
|
||||
already wrote into these repositories and needs explaining.
|
||||
- [ ] Workspaces reporting `0 file(s)` are genuinely empty (cross-check against 2.4).
|
||||
|
||||
**Stop if** any workspace errors, or `extra`/`mismatched` is non-zero.
|
||||
|
||||
---
|
||||
|
||||
## Stage 4 — Seed for real (still inert)
|
||||
|
||||
Seeding is safe to run while the site is live and taking writes, because with the flags off no other
|
||||
writer touches git: the recorder and the turn-commit path both no-op for unflipped workspaces. There
|
||||
is no lock contention to fear yet.
|
||||
|
||||
**4a. Canary set first** — the small internal workspaces from 2.4:
|
||||
|
||||
```bash
|
||||
cd /app && python scripts/migrate_knowledge_store.py --yes \
|
||||
--workspace <A> --workspace <B> \
|
||||
--out /app/.local_object_store/ks-migration/seed-canary.jsonl
|
||||
```
|
||||
|
||||
**4b. Then the fleet**, once the canary reports `ok`:
|
||||
|
||||
```bash
|
||||
cd /app && python scripts/migrate_knowledge_store.py --yes \
|
||||
--out /app/.local_object_store/ks-migration/seed-fleet.jsonl
|
||||
```
|
||||
|
||||
What one workspace's seed does: reads each document's `source_markdown` (falling back to `content`,
|
||||
skipping blanks and `Pending...`), resolves a path per row (recorded marker first, title-derived
|
||||
otherwise), writes them all as **one** revision authored by the migration identity, removes any
|
||||
tracked path not in the desired set so a re-run converges, verifies parity by content address, then
|
||||
stamps `virtual_path` back onto each row.
|
||||
|
||||
**Cost model.** No embeddings, no model calls, no re-chunking — the seed copies bytes, which is the
|
||||
whole point of "adopt, don't rebuild". Runtime is dominated by writing git objects for the corpus
|
||||
measured in 1.5, plus one metadata `UPDATE` per document.
|
||||
|
||||
**Checks**
|
||||
|
||||
- [ ] Every line reads `ok, N file(s)`; the summary reads `seeded: X ok, 0 failed of X`.
|
||||
- [ ] No `Could not record seeded paths` in the logs (a marker failure leaves rows that cannot
|
||||
survive a retitle; re-running the seed repairs it).
|
||||
- [ ] Marker coverage matches the seeded count:
|
||||
```sql
|
||||
SELECT count(*) FROM documents
|
||||
WHERE document_metadata::jsonb ->> 'virtual_path' IS NOT NULL;
|
||||
```
|
||||
- [ ] Row counts unchanged from stage 2 (seeding must not create or delete documents):
|
||||
```sql
|
||||
SELECT count(*) FROM documents;
|
||||
```
|
||||
- [ ] Disk grew by roughly the predicted amount, and free space is still comfortable.
|
||||
|
||||
**Stop if** any workspace fails. Re-running is idempotent and convergent, so a partial pass is safe
|
||||
to resume — but understand *why* it failed first.
|
||||
|
||||
---
|
||||
|
||||
## Stage 5 — Verify parity
|
||||
|
||||
```bash
|
||||
cd /app && python scripts/migrate_knowledge_store.py \
|
||||
--out /app/.local_object_store/ks-migration/verify-$(date -u +%Y%m%dT%H%M%SZ).jsonl
|
||||
```
|
||||
|
||||
- [ ] Every workspace reads `ok`, and the command **exits 0**.
|
||||
- [ ] Spot-check a handful of documents by hand: read the blob at head and diff it against
|
||||
`source_markdown` for the same row. Byte identity is the seed's whole claim.
|
||||
|
||||
**Stop if** anything is not `ok`. Do not flip a workspace whose parity fails — the drift monitor
|
||||
would auto-enqueue a full re-embed for it (capped at 10 workspaces per nightly run), which is
|
||||
exactly the cost this migration exists to avoid.
|
||||
|
||||
---
|
||||
|
||||
## Stage 6 — Turn on the global flag (still nothing flipped)
|
||||
|
||||
Set `KNOWLEDGE_STORE_ENABLED=TRUE` on **api** and **worker** (beat is harmless either way) and
|
||||
redeploy those services. Nothing changes behaviour yet, because every workspace's column is still
|
||||
false — this stage exists so that the flip in stage 7 is a single, reversible database write rather
|
||||
than a deploy.
|
||||
|
||||
**Checks**
|
||||
|
||||
- [ ] `env | grep KNOWLEDGE_STORE_ENABLED` shows TRUE in api and worker.
|
||||
- [ ] A chat turn on an unflipped workspace still behaves exactly as before (the compiled agent
|
||||
cache key includes the per-workspace flag, so no stale graph is served).
|
||||
- [ ] Nothing new appeared under `/app/.local_object_store/knowledge_store/*/`.
|
||||
|
||||
---
|
||||
|
||||
## Stage 7 — Flip, in batches
|
||||
|
||||
```bash
|
||||
cd /app && python scripts/migrate_knowledge_store.py --yes --flip \
|
||||
--workspace <A> \
|
||||
--out /app/.local_object_store/ks-migration/flip-A.jsonl
|
||||
```
|
||||
|
||||
`--flip` refuses to run without `--yes`, only flips a workspace whose parity passed in the same
|
||||
pass, and stamps `last_indexed_revision` to the store's head as it goes. That stamp is load-bearing:
|
||||
leave it NULL and the hourly sweep reads the workspace as never-indexed and re-embeds the entire
|
||||
tree.
|
||||
|
||||
**Immediately after the first flip, verify by hand**
|
||||
|
||||
- [ ] `SELECT id, knowledge_store_enabled, last_indexed_revision FROM workspaces WHERE id = <A>;`
|
||||
and the stamp equals the repository's head.
|
||||
- [ ] One agent turn that writes a note: a new revision appears, and the document row appears in the
|
||||
UI with the right title and folder.
|
||||
- [ ] One editor save on that note: it records, and the title is not silently renamed.
|
||||
- [ ] Search returns seeded content for that workspace.
|
||||
|
||||
**Then watch the clock** (all UTC, `app/celery_app.py:345-366`):
|
||||
|
||||
- **:20 every hour** — the sweep. For a correctly stamped workspace it should be a **no-op**. If it
|
||||
re-indexes the whole tree, the stamp was wrong; stop flipping.
|
||||
- **04:45 daily** — the working-copy janitor.
|
||||
- **05:15 daily** — the drift monitor. Expect `status=ok`. This is the strongest single signal that a
|
||||
flip is healthy.
|
||||
|
||||
**Batch size.** Flip in groups of **≤ 10**, and let one nightly drift check pass between groups. Ten
|
||||
matches the monitor's per-run repair cap, so if a whole batch goes wrong, one night's auto-repair can
|
||||
cover it.
|
||||
|
||||
---
|
||||
|
||||
## Stage 8 — Watching, and rolling back
|
||||
|
||||
### What to watch
|
||||
|
||||
Metrics exist but only when an OTLP endpoint is configured; otherwise they no-op silently
|
||||
(`app/observability/otel.py:62-84`), so **logs are the primary signal** unless you wire a collector:
|
||||
|
||||
| Signal | Where |
|
||||
|---|---|
|
||||
| `surfsense.knowledge_store.drift.check` counter, labels `workspace.id`, `status` | metrics, if OTLP configured |
|
||||
| `surfsense.knowledge_store.record.outcome` counter, labels `flow` (`editor_save`/`sync_batch`/`turn_commit`), `status` (`recorded`/`noop`/`failed`) | metrics, if OTLP configured |
|
||||
| `Knowledge store drift check for workspace %s: %s (missing=… extra=… mismatched=…)` | worker log, nightly |
|
||||
| `Knowledge store index for workspace %s: revision=… indexed=… skipped=… failed=… deleted=… stamped=…` | worker log, per index run |
|
||||
| `Could not acquire index_lock/write_lock for workspace …` | worker log — contention or a leaked lock |
|
||||
| `Knowledge store recording failed for document %s in workspace %s` | api log — an editor save that did not reach git |
|
||||
| `End-of-turn commit failed for workspace %s thread %s` | worker log — a turn whose writes were kept for next-turn recovery |
|
||||
|
||||
Alarm-worthy: any `failed=` other than zero in an index outcome, any `status=drift` after the first
|
||||
night, and any lock error that repeats.
|
||||
|
||||
### Rolling back
|
||||
|
||||
- **One workspace:**
|
||||
```bash
|
||||
cd /app && python scripts/migrate_knowledge_store.py --unflip --workspace <A>
|
||||
```
|
||||
Clears the column and the stamp — deliberately, so a later re-flip does a full reconcile, since
|
||||
the legacy pipeline owns the chunks in between.
|
||||
- **The whole fleet, immediately:** set `KNOWLEDGE_STORE_ENABLED=FALSE` and redeploy api + worker. No
|
||||
database write, no per-workspace bookkeeping.
|
||||
- **What rollback does not undo:** revisions committed to git while the workspace was flipped stay in
|
||||
git, and the rows the indexer wrote stay in Postgres. That is harmless — the content is the same
|
||||
content — but re-flipping later should be treated as a fresh seed-and-verify.
|
||||
|
||||
---
|
||||
|
||||
## Known gaps to accept (or close) before flipping a UI-live workspace
|
||||
|
||||
These are tracked in the phase plans, not defects introduced by the migration:
|
||||
|
||||
1. **Phase 6 — the SSE channel.** Only the legacy middleware emits `document_created`/`updated`/
|
||||
`deleted`/`folder_deleted`; nothing on the git-native path does, so the in-chat document cards
|
||||
stop appearing for a flipped workspace. Zero replication of the rows still works, so the document
|
||||
list itself stays live, with a lag equal to the index queue.
|
||||
2. **`DELETE /documents/{id}`** removes the row but not the git file, so the drift monitor resurrects
|
||||
the document.
|
||||
3. **`PUT /documents/{id}`** retitles the row but leaves its marker and identity hash on the old
|
||||
path.
|
||||
4. **Cosmetic:** storage paths still carry `.xml` for anything named from a title. Harmless, and
|
||||
retiring it is deferred.
|
||||
|
||||
Items 2 and 3 are the two that a normal user can trigger from the UI, so close them before flipping
|
||||
a workspace with real users, or accept the behaviour knowingly.
|
||||
|
||||
---
|
||||
|
||||
## Appendix — one-line summary of each command
|
||||
|
||||
| Purpose | Command |
|
||||
|---|---|
|
||||
| Dry run, whole fleet | `python scripts/migrate_knowledge_store.py --out <report>` |
|
||||
| Dry run, one workspace | `… --workspace <id>` |
|
||||
| Seed for real | `… --yes` |
|
||||
| Seed + flip | `… --yes --flip --workspace <id>` |
|
||||
| Roll one workspace back | `… --unflip --workspace <id>` |
|
||||
| Force a full reindex of one workspace | enqueue `reindex_knowledge_store` for that id |
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
# Phase 6 — Zero / real-time projection
|
||||
|
||||
> Depends on Phases 3/4. The one genuinely net-new subsystem (partly offsets the deletions). Umbrella: [`00-umbrella-plan.md`](00-umbrella-plan.md).
|
||||
|
||||
## Objective
|
||||
|
||||
Keep the real-time web UI working after git becomes the source of truth. The UI is driven by Zero (Postgres logical replication, `zero_publication.py`); git is not a real-time source, so we project git state → the Zero-published `documents`/`folders` rows after each commit.
|
||||
|
||||
## Locked model
|
||||
|
||||
- **Two UI channels, both must survive** (see [`00c-shared-contract.md`](00c-shared-contract.md) C5): (1) Zero logical replication of the `documents`/`folders` rows (`zero_publication.py`); (2) the `dispatch_custom_event` SSE (`document_created/updated/deleted`, `folder_deleted`) from the commit path. The projection must keep the rows current **and** the events must keep firing.
|
||||
- **One-way projection (git → Postgres rows), not a second source of truth.** The projected `documents`/`folders` rows are **thin metadata for the UI** (title, path, folder tree, timestamps) — content authority stays in git; chunks/embeddings stay derived (Phase 4). This preserves the "no two-way sync" rule.
|
||||
- **Driven by the commit event** (Phase 3). Simplest owner: extend the Phase-4 post-commit indexer to also upsert/delete the Zero-published rows in the same pass (index + project together).
|
||||
- Zero publication column lists (`zero_publication.py`) stay as-is; we just keep the rows current from git.
|
||||
|
||||
## Work items
|
||||
|
||||
1. `project_commit(workspace_id, sha)` — from the tree diff, upsert `documents`/`folders` metadata rows and delete rows for removed paths; publish via the existing Zero path.
|
||||
2. Decide owner: fold into Phase-4 `index_commit` (one post-commit pass) vs. a separate projector task — default: fold in.
|
||||
3. Ensure ordering: commit → index + project atomically enough that the UI never shows a file with no metadata row (or vice versa).
|
||||
|
||||
## Tests
|
||||
|
||||
- After a commit, Zero-published `documents`/`folders` rows reflect the new tree within the projection cycle.
|
||||
- Deleting a file removes its row; renaming updates path/title.
|
||||
- A workspace with the flag on stays live in the web UI end-to-end (create/edit/delete visible).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Replacing Zero with a git-aware sync (not now).
|
||||
- Frontend changes beyond keeping current behavior (separate umbrella).
|
||||
|
||||
## Resolved (see [`00c-shared-contract.md`](00c-shared-contract.md))
|
||||
|
||||
- **Projection owner:** fold into the Phase-4 post-commit indexer (index + project one pass) (C5).
|
||||
- **Both channels preserved:** upsert/delete `documents`/`folders` rows for Zero **and** keep emitting the SSE custom events (C5).
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Consistency model: index + projection in one pass, or eventually consistent with a short lag (default: one pass).
|
||||
2. Do any Zero-published columns need content that isn't cheap to derive from the tree (forces a richer projection)?
|
||||
|
|
@ -468,18 +468,6 @@ FILE_STORAGE_BACKEND=local
|
|||
# AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net
|
||||
# AZURE_STORAGE_CONTAINER=surfsense-documents
|
||||
|
||||
# Knowledge Store (Git-native KB)
|
||||
# Makes a workspace's documents a git repository: every save, connector sync and
|
||||
# agent turn is a commit, and the Postgres chunk rows become a derived index
|
||||
# rebuilt from it. This is the master kill switch only — a workspace goes
|
||||
# git-native when this AND workspaces.knowledge_store_enabled are both on, and
|
||||
# scripts/migrate_knowledge_store.py sets the latter once its seed passes byte
|
||||
# parity. Turning this off reverts every workspace to the Postgres write path.
|
||||
KNOWLEDGE_STORE_ENABLED=FALSE
|
||||
# Where the per-workspace repositories live (defaults to <file storage>/knowledge_store).
|
||||
# Must be a shared volume: every web and worker process needs the same history.
|
||||
# KNOWLEDGE_STORE_ROOT=/var/lib/surfsense/object-store/knowledge_store
|
||||
|
||||
# ETL Parse Cache
|
||||
# Reuse parser output for identical file bytes across workspaces (skips paid
|
||||
# re-parsing on LlamaCloud / Azure DI / Unstructured). Off by default.
|
||||
|
|
|
|||
1
surfsense_backend/.gitignore
vendored
1
surfsense_backend/.gitignore
vendored
|
|
@ -16,7 +16,6 @@ celerybeat-schedule.dir
|
|||
celerybeat-schedule.bak
|
||||
/app/config/global_llm_config.yaml
|
||||
app/templates/_generated/
|
||||
knowledge_store_migration_reports.jsonl
|
||||
|
||||
/tests/unit/platforms/instagram/fixtures/post.json
|
||||
/tests/unit/platforms/instagram/fixtures/profile.json
|
||||
|
|
|
|||
|
|
@ -1,31 +0,0 @@
|
|||
"""Add workspaces.knowledge_store_enabled for the progressive git-native flip.
|
||||
|
||||
Per-workspace switch between the old write path and the git-native one; the
|
||||
global KNOWLEDGE_STORE_ENABLED env stays the master kill switch (a workspace
|
||||
is git-native only when both are on). Default false: every workspace keeps
|
||||
the old path until its seed passes parity and it is flipped explicitly.
|
||||
|
||||
Revision ID: 175
|
||||
Revises: 174
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "175"
|
||||
down_revision: str | None = "174"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
"ALTER TABLE workspaces "
|
||||
"ADD COLUMN IF NOT EXISTS knowledge_store_enabled BOOLEAN "
|
||||
"NOT NULL DEFAULT FALSE"
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("ALTER TABLE workspaces DROP COLUMN IF EXISTS knowledge_store_enabled")
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
"""Add the derived-index columns: workspace drift marker + chunk line spans.
|
||||
|
||||
``workspaces.last_indexed_revision`` records which store revision the chunk
|
||||
index was built from; NULL means never indexed, which is exactly what makes the
|
||||
drift sweep pick a workspace up, so no backfill is wanted.
|
||||
|
||||
``chunks.start_line`` / ``chunks.end_line`` are the 1-based inclusive line range
|
||||
each chunk was cut from. NULL on existing rows: they are re-populated the next
|
||||
time their document is indexed, and their only consumer (line numbers on search
|
||||
excerpts) treats NULL as "no line info".
|
||||
|
||||
One migration for both tables so the two columns cannot become two Alembic
|
||||
heads.
|
||||
|
||||
Revision ID: 176
|
||||
Revises: 175
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "176"
|
||||
down_revision: str | None = "175"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
"ALTER TABLE workspaces "
|
||||
"ADD COLUMN IF NOT EXISTS last_indexed_revision VARCHAR(64)"
|
||||
)
|
||||
op.execute("ALTER TABLE chunks ADD COLUMN IF NOT EXISTS start_line INTEGER")
|
||||
op.execute("ALTER TABLE chunks ADD COLUMN IF NOT EXISTS end_line INTEGER")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.execute("ALTER TABLE chunks DROP COLUMN IF EXISTS end_line")
|
||||
op.execute("ALTER TABLE chunks DROP COLUMN IF EXISTS start_line")
|
||||
op.execute("ALTER TABLE workspaces DROP COLUMN IF EXISTS last_indexed_revision")
|
||||
|
|
@ -45,7 +45,6 @@ def build_compiled_agent_graph_sync(
|
|||
subagent_dependencies: dict[str, Any],
|
||||
mcp_tools_by_agent: dict[str, list[BaseTool]] | None = None,
|
||||
disabled_tools: list[str] | None = None,
|
||||
knowledge_store_enabled: bool = False,
|
||||
):
|
||||
"""Sync compile: middleware + ``create_agent`` (run via ``asyncio.to_thread``)."""
|
||||
mw_start = time.perf_counter()
|
||||
|
|
@ -68,7 +67,6 @@ def build_compiled_agent_graph_sync(
|
|||
checkpointer=checkpointer,
|
||||
mcp_tools_by_agent=mcp_tools_by_agent,
|
||||
disabled_tools=disabled_tools,
|
||||
knowledge_store_enabled=knowledge_store_enabled,
|
||||
)
|
||||
mw_elapsed = time.perf_counter() - mw_start
|
||||
|
||||
|
|
|
|||
|
|
@ -80,7 +80,7 @@ def _basename(path: str) -> str:
|
|||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def ensure_folder_hierarchy(
|
||||
async def _ensure_folder_hierarchy(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
workspace_id: int,
|
||||
|
|
@ -190,7 +190,7 @@ async def _create_document(
|
|||
folder_parts, title = parse_documents_path(virtual_path)
|
||||
if not title:
|
||||
raise ValueError(f"invalid /documents path '{virtual_path}'")
|
||||
folder_id = await ensure_folder_hierarchy(
|
||||
folder_id = await _ensure_folder_hierarchy(
|
||||
session,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=created_by_id,
|
||||
|
|
@ -360,7 +360,7 @@ async def _apply_move(
|
|||
folder_parts, new_title = parse_documents_path(dest)
|
||||
if not new_title:
|
||||
return None
|
||||
folder_id = await ensure_folder_hierarchy(
|
||||
folder_id = await _ensure_folder_hierarchy(
|
||||
session,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=created_by_id,
|
||||
|
|
@ -810,7 +810,7 @@ async def commit_staged_filesystem_state(
|
|||
folder_parts_full = _split_folder_path(folder_path)
|
||||
if not folder_parts_full:
|
||||
continue
|
||||
folder_id = await ensure_folder_hierarchy(
|
||||
folder_id = await _ensure_folder_hierarchy(
|
||||
session,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=created_by_id,
|
||||
|
|
|
|||
|
|
@ -1,11 +0,0 @@
|
|||
"""Git-native end-of-turn persistence: one revision per agent turn."""
|
||||
|
||||
from .builder import build_knowledge_store_persistence_mw
|
||||
from .commit_turn import commit_turn_working_copy
|
||||
from .middleware import KnowledgeStorePersistenceMiddleware
|
||||
|
||||
__all__ = [
|
||||
"KnowledgeStorePersistenceMiddleware",
|
||||
"build_knowledge_store_persistence_mw",
|
||||
"commit_turn_working_copy",
|
||||
]
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
"""Build the git-native persistence middleware when the flag selects it."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.filesystem_selection import FilesystemMode
|
||||
|
||||
from .middleware import KnowledgeStorePersistenceMiddleware
|
||||
|
||||
|
||||
def build_knowledge_store_persistence_mw(
|
||||
*,
|
||||
filesystem_mode: FilesystemMode,
|
||||
workspace_id: int,
|
||||
user_id: str | None,
|
||||
thread_id: int | None,
|
||||
llm: Any,
|
||||
knowledge_store_enabled: bool = False,
|
||||
) -> KnowledgeStorePersistenceMiddleware | None:
|
||||
"""``knowledge_store_enabled`` is the caller's once-per-turn verdict
|
||||
(global switch AND workspace flip); defaults off, the safe path."""
|
||||
if filesystem_mode != FilesystemMode.CLOUD:
|
||||
return None
|
||||
if not knowledge_store_enabled:
|
||||
return None
|
||||
return KnowledgeStorePersistenceMiddleware(
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=user_id,
|
||||
thread_id=thread_id,
|
||||
llm=llm,
|
||||
)
|
||||
|
|
@ -1,81 +0,0 @@
|
|||
"""Commit messages for a turn's revision: model-generated subject, deterministic fallback."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Iterable, Mapping
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SYSTEM_PROMPT = (
|
||||
"You write git commit messages for a knowledge-base revision. "
|
||||
"Reply with ONE line only: an imperative subject in Conventional Commits "
|
||||
"style (e.g. 'docs: add meeting notes'). No quotes, no body, no trailers."
|
||||
)
|
||||
|
||||
# ponytail: previews cap prompt size; a huge turn still gets a decent subject
|
||||
# from the first files alone. Upgrade path: real unified diffs.
|
||||
_MAX_FILES_IN_PROMPT = 20
|
||||
_MAX_PREVIEW_CHARS = 300
|
||||
|
||||
# The subject is a nicety on the path that ends a turn: the commit, the working
|
||||
# copy's discard and the turn's outcome all wait behind it. A provider that
|
||||
# accepts the request and then stalls never raises, so the deadline — not the
|
||||
# ``except`` below — is what keeps a hung generation from stranding the write.
|
||||
_GENERATION_TIMEOUT_SECONDS = 10.0
|
||||
|
||||
|
||||
def fallback_commit_message(
|
||||
*, writes: Mapping[str, bytes], removes: Iterable[str]
|
||||
) -> str:
|
||||
"""Deterministic subject used whenever generation fails; never raises."""
|
||||
removed = list(removes)
|
||||
parts: list[str] = []
|
||||
if writes:
|
||||
parts.append(f"update {len(writes)} file(s)")
|
||||
if removed:
|
||||
parts.append(f"remove {len(removed)} file(s)")
|
||||
return "chore: " + ", ".join(parts)
|
||||
|
||||
|
||||
def _describe_changes(writes: Mapping[str, bytes], removes: Iterable[str]) -> str:
|
||||
lines: list[str] = []
|
||||
for path, content in list(writes.items())[:_MAX_FILES_IN_PROMPT]:
|
||||
preview = content[:_MAX_PREVIEW_CHARS].decode("utf-8", errors="replace")
|
||||
lines.append(f"WRITE {path}\n{preview}")
|
||||
for path in list(removes)[:_MAX_FILES_IN_PROMPT]:
|
||||
lines.append(f"REMOVE {path}")
|
||||
return "\n\n".join(lines)
|
||||
|
||||
|
||||
async def generate_commit_message(
|
||||
llm: Any | None, *, writes: Mapping[str, bytes], removes: Iterable[str]
|
||||
) -> str:
|
||||
"""One-line subject for the turn's revision. Falls back rather than raise or
|
||||
hang: a commit must never be lost to message generation. ``llm=None`` (the
|
||||
disconnect fallback path) uses the deterministic subject directly."""
|
||||
if llm is None:
|
||||
return fallback_commit_message(writes=writes, removes=removes)
|
||||
try:
|
||||
reply = await asyncio.wait_for(
|
||||
llm.ainvoke(
|
||||
[
|
||||
("system", _SYSTEM_PROMPT),
|
||||
("human", _describe_changes(writes, removes)),
|
||||
]
|
||||
),
|
||||
timeout=_GENERATION_TIMEOUT_SECONDS,
|
||||
)
|
||||
content = getattr(reply, "content", "")
|
||||
if not isinstance(content, str):
|
||||
content = str(content)
|
||||
subject = content.strip().splitlines()[0].strip() if content.strip() else ""
|
||||
if subject:
|
||||
return subject
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"Commit message generation failed; using fallback", exc_info=True
|
||||
)
|
||||
return fallback_commit_message(writes=writes, removes=removes)
|
||||
|
|
@ -1,128 +0,0 @@
|
|||
"""End-of-turn commit body: the turn's working copy becomes one revision.
|
||||
|
||||
A free function (not a middleware method) so the stream-task disconnect
|
||||
fallback can run the identical routine when ``aafter_agent`` is skipped.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Iterable, Mapping
|
||||
from typing import Any
|
||||
|
||||
from app.agents.chat.multi_agent_chat.main_agent.middleware.knowledge_store_persistence.commit_message import (
|
||||
generate_commit_message,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.git_tree import (
|
||||
thread_working_copy_id,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.receipts.receipt import (
|
||||
Receipt,
|
||||
make_receipt,
|
||||
)
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.knowledge_store.identities import AGENT_IDENTITY, user_identity
|
||||
from app.knowledge_store.index.queue import enqueue_index
|
||||
from app.observability import metrics
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_OPERATION_BY_KIND = {
|
||||
"added": "write_file",
|
||||
"modified": "edit_file",
|
||||
"removed": "rm",
|
||||
"renamed": "move_file",
|
||||
}
|
||||
|
||||
|
||||
async def commit_turn_working_copy(
|
||||
*,
|
||||
workspace_id: int | str,
|
||||
thread_id: int | str | None,
|
||||
created_by_id: str | None,
|
||||
llm: Any,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Record the thread's working copy as one revision; return the state delta.
|
||||
|
||||
No copy or an empty diff records nothing. On commit failure the copy is
|
||||
kept — the thread's next turn commits the leftover work (recovery) — and
|
||||
failed receipts are returned instead of raising, so the turn still ends.
|
||||
"""
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
copy_id = thread_working_copy_id(thread_id)
|
||||
try:
|
||||
writes, removes = await store.diff_working_copy(copy_id)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
if not writes and not removes:
|
||||
await store.discard_working_copy(copy_id)
|
||||
return None
|
||||
|
||||
subject = await generate_commit_message(llm, writes=writes, removes=removes)
|
||||
message = f"{subject}\n\nThread: {thread_id}"
|
||||
try:
|
||||
async with store.transaction(
|
||||
message=message,
|
||||
author=user_identity(created_by_id),
|
||||
committer=AGENT_IDENTITY,
|
||||
) as tx:
|
||||
for path, content in writes.items():
|
||||
tx.write(path, content)
|
||||
for path in removes:
|
||||
tx.remove(path)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"End-of-turn commit failed for workspace %s thread %s: %s",
|
||||
workspace_id,
|
||||
thread_id,
|
||||
exc,
|
||||
)
|
||||
metrics.record_knowledge_store_record_outcome(
|
||||
flow="turn_commit",
|
||||
status="failed",
|
||||
error_category=metrics.categorize_exception(exc),
|
||||
)
|
||||
return {"receipts": _failed_receipts(writes, removes, exc)}
|
||||
|
||||
await store.discard_working_copy(copy_id)
|
||||
metrics.record_knowledge_store_record_outcome(
|
||||
flow="turn_commit", status="recorded" if tx.revision else "noop"
|
||||
)
|
||||
if tx.revision is None:
|
||||
return None
|
||||
enqueue_index(workspace_id)
|
||||
return {"receipts": await _recorded_receipts(store, tx.revision)}
|
||||
|
||||
|
||||
async def _recorded_receipts(store: KnowledgeStore, revision: str) -> list[Receipt]:
|
||||
"""Ground truth for the orchestrator: one receipt per recorded change."""
|
||||
return [
|
||||
make_receipt(
|
||||
route="knowledge_base",
|
||||
type="file",
|
||||
operation=_OPERATION_BY_KIND[change.kind],
|
||||
status="success",
|
||||
external_id=revision,
|
||||
preview=change.path,
|
||||
)
|
||||
for change in await store.list_changes(revision)
|
||||
]
|
||||
|
||||
|
||||
def _failed_receipts(
|
||||
writes: Mapping[str, bytes], removes: Iterable[str], exc: Exception
|
||||
) -> list[Receipt]:
|
||||
return [
|
||||
make_receipt(
|
||||
route="knowledge_base",
|
||||
type="file",
|
||||
operation="write_file" if is_write else "rm",
|
||||
status="failed",
|
||||
preview=path,
|
||||
error=str(exc),
|
||||
)
|
||||
for path, is_write in (
|
||||
*((p, True) for p in writes),
|
||||
*((p, False) for p in removes),
|
||||
)
|
||||
]
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
"""End-of-turn hook: commit the turn's working copy as one revision."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain.agents.middleware import AgentMiddleware, AgentState
|
||||
from langgraph.config import get_config
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
from app.agents.chat.multi_agent_chat.main_agent.middleware.knowledge_store_persistence.commit_turn import (
|
||||
commit_turn_working_copy,
|
||||
)
|
||||
|
||||
|
||||
class KnowledgeStorePersistenceMiddleware(AgentMiddleware): # type: ignore[type-arg]
|
||||
"""Runs the commit body after the agent's turn (git-native write path)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
workspace_id: int,
|
||||
created_by_id: str | None,
|
||||
thread_id: int | None,
|
||||
llm: Any,
|
||||
) -> None:
|
||||
self.workspace_id = workspace_id
|
||||
self.created_by_id = created_by_id
|
||||
self.thread_id = thread_id
|
||||
self.llm = llm
|
||||
|
||||
async def aafter_agent( # type: ignore[override]
|
||||
self,
|
||||
state: AgentState,
|
||||
runtime: Runtime[Any],
|
||||
) -> dict[str, Any] | None:
|
||||
del state, runtime # the working copy on disk is the pending state
|
||||
return await commit_turn_working_copy(
|
||||
workspace_id=self.workspace_id,
|
||||
thread_id=self._resolve_thread_id(),
|
||||
created_by_id=self.created_by_id,
|
||||
llm=self.llm,
|
||||
)
|
||||
|
||||
def _resolve_thread_id(self) -> int | str | None:
|
||||
"""Live thread id from the active config, so one cached compiled graph
|
||||
commits against the correct thread across many chats (same pattern as
|
||||
``kb_persistence``)."""
|
||||
try:
|
||||
config = get_config()
|
||||
except Exception:
|
||||
config = None
|
||||
if isinstance(config, dict):
|
||||
value = (config.get("configurable") or {}).get("thread_id")
|
||||
if value is not None:
|
||||
return value
|
||||
return self.thread_id
|
||||
|
|
@ -41,7 +41,7 @@ from app.agents.chat.runtime.path_resolver import (
|
|||
DOCUMENTS_ROOT,
|
||||
PathIndex,
|
||||
build_path_index,
|
||||
virtual_path_of,
|
||||
doc_to_virtual_path,
|
||||
)
|
||||
from app.db import Document, shielded_async_session
|
||||
from app.utils.perf import get_perf_logger
|
||||
|
|
@ -199,12 +199,9 @@ class KnowledgeTreeMiddleware(AgentMiddleware): # type: ignore[type-arg]
|
|||
async with shielded_async_session() as session:
|
||||
index = await build_path_index(session, self.workspace_id)
|
||||
doc_rows = await session.execute(
|
||||
select(
|
||||
Document.id,
|
||||
Document.title,
|
||||
Document.folder_id,
|
||||
Document.document_metadata,
|
||||
).where(Document.workspace_id == self.workspace_id)
|
||||
select(Document.id, Document.title, Document.folder_id).where(
|
||||
Document.workspace_id == self.workspace_id
|
||||
)
|
||||
)
|
||||
docs = list(doc_rows.all())
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
|
|
@ -218,8 +215,7 @@ class KnowledgeTreeMiddleware(AgentMiddleware): # type: ignore[type-arg]
|
|||
def _format_tree(self, index: PathIndex, docs: list[Any]) -> str:
|
||||
folder_paths = sorted(set(index.folder_paths.values()))
|
||||
doc_paths = sorted(
|
||||
virtual_path_of(
|
||||
metadata=row.document_metadata,
|
||||
doc_to_virtual_path(
|
||||
doc_id=row.id,
|
||||
title=str(row.title or "untitled"),
|
||||
folder_id=row.folder_id,
|
||||
|
|
|
|||
|
|
@ -88,7 +88,6 @@ from .context_editing import build_context_editing_mw
|
|||
from .dedup_hitl import build_dedup_hitl_mw
|
||||
from .doom_loop import build_doom_loop_mw
|
||||
from .kb_persistence import build_kb_persistence_mw
|
||||
from .knowledge_store_persistence import build_knowledge_store_persistence_mw
|
||||
from .knowledge_tree import build_knowledge_tree_mw
|
||||
from .noop_injection import build_noop_injection_mw
|
||||
from .otel_span import build_otel_mw
|
||||
|
|
@ -119,7 +118,6 @@ def build_main_agent_deepagent_middleware(
|
|||
checkpointer: Checkpointer,
|
||||
mcp_tools_by_agent: dict[str, list[BaseTool]] | None = None,
|
||||
disabled_tools: list[str] | None = None,
|
||||
knowledge_store_enabled: bool = False,
|
||||
) -> list[Any]:
|
||||
"""Ordered middleware for ``create_agent`` (None entries already stripped)."""
|
||||
stack_build_start = time.perf_counter()
|
||||
|
|
@ -136,7 +134,6 @@ def build_main_agent_deepagent_middleware(
|
|||
"backend_resolver": backend_resolver,
|
||||
"filesystem_mode": filesystem_mode,
|
||||
"flags": flags,
|
||||
"knowledge_store_enabled": knowledge_store_enabled,
|
||||
}
|
||||
shared_mw_start = time.perf_counter()
|
||||
shared_subagent_middleware = build_subagent_middleware_stack(
|
||||
|
|
@ -249,17 +246,6 @@ def build_main_agent_deepagent_middleware(
|
|||
user_id=user_id,
|
||||
thread_id=thread_id,
|
||||
),
|
||||
# Git-native write path; coexists with kb_persistence until the
|
||||
# Phase 5 cut (unflagged workspaces still stage state, flagged ones
|
||||
# leave it empty so the old commit body no-ops).
|
||||
build_knowledge_store_persistence_mw(
|
||||
filesystem_mode=filesystem_mode,
|
||||
workspace_id=workspace_id,
|
||||
user_id=user_id,
|
||||
thread_id=thread_id,
|
||||
llm=llm,
|
||||
knowledge_store_enabled=knowledge_store_enabled,
|
||||
),
|
||||
build_skills_mw(
|
||||
flags=flags,
|
||||
filesystem_mode=filesystem_mode,
|
||||
|
|
|
|||
|
|
@ -58,7 +58,6 @@ async def build_agent_with_cache(
|
|||
disabled_tools: list[str] | None,
|
||||
config_id: str | None,
|
||||
image_gen_model_id_override: int | None = None,
|
||||
knowledge_store_enabled: bool = False,
|
||||
) -> Any:
|
||||
"""Compile the multi-agent graph, serving from cache when key components are stable."""
|
||||
|
||||
|
|
@ -84,7 +83,6 @@ async def build_agent_with_cache(
|
|||
subagent_dependencies=subagent_dependencies,
|
||||
mcp_tools_by_agent=mcp_tools_by_agent,
|
||||
disabled_tools=disabled_tools,
|
||||
knowledge_store_enabled=knowledge_store_enabled,
|
||||
)
|
||||
|
||||
if not (flags.enable_agent_cache and not flags.disable_new_agent_stack):
|
||||
|
|
@ -124,10 +122,6 @@ async def build_agent_with_cache(
|
|||
# must key the compiled-agent cache to avoid leaking one automation's
|
||||
# image model into another with the same config_id/workspace.
|
||||
image_gen_model_id_override,
|
||||
# Selects the filesystem backend, the persistence middleware, and
|
||||
# ``read_file``'s description at build time, so a workspace flip must
|
||||
# rotate the cached graph.
|
||||
knowledge_store_enabled,
|
||||
)
|
||||
return await get_cache().get_or_build(cache_key, builder=_build)
|
||||
|
||||
|
|
|
|||
|
|
@ -36,7 +36,6 @@ from app.agents.chat.runtime.prompt_caching import (
|
|||
)
|
||||
from app.auth.context import AuthContext
|
||||
from app.db import ChatVisibility
|
||||
from app.knowledge_store.settings import knowledge_store_enabled_for
|
||||
from app.services.connector_service import ConnectorService
|
||||
from app.services.user_tool_allowlist import (
|
||||
fetch_user_allowlist_rulesets,
|
||||
|
|
@ -87,18 +86,11 @@ async def create_multi_agent_chat_deep_agent(
|
|||
apply_litellm_prompt_caching(llm, agent_config=agent_config, thread_id=thread_id)
|
||||
|
||||
filesystem_selection = filesystem_selection or FilesystemSelection()
|
||||
# Resolved once here; the whole turn (backend, middleware, cached graph)
|
||||
# inherits this verdict, so a mid-turn flip can't mix write paths.
|
||||
git_native = (
|
||||
filesystem_selection.mode == FilesystemMode.CLOUD
|
||||
and await knowledge_store_enabled_for(workspace_id)
|
||||
)
|
||||
backend_resolver = build_backend_resolver(
|
||||
filesystem_selection,
|
||||
workspace_id=workspace_id
|
||||
if filesystem_selection.mode == FilesystemMode.CLOUD
|
||||
else None,
|
||||
knowledge_store_enabled=git_native,
|
||||
)
|
||||
|
||||
available_connectors: list[str] | None = None
|
||||
|
|
@ -318,7 +310,6 @@ async def create_multi_agent_chat_deep_agent(
|
|||
disabled_tools=disabled_tools,
|
||||
config_id=config_id,
|
||||
image_gen_model_id_override=image_gen_model_id,
|
||||
knowledge_store_enabled=git_native,
|
||||
)
|
||||
_perf_log.info(
|
||||
"[create_agent] Middleware stack + graph compiled in %.3fs",
|
||||
|
|
|
|||
|
|
@ -1,160 +0,0 @@
|
|||
"""deepagents adapter: agent file ops on the turn's private working copy."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from deepagents.backends.protocol import (
|
||||
EditResult,
|
||||
FileDownloadResponse,
|
||||
FileInfo,
|
||||
FileUploadResponse,
|
||||
GrepMatch,
|
||||
WriteResult,
|
||||
)
|
||||
from langgraph.prebuilt.tool_node import ToolRuntime
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.multi_root_local_folder import (
|
||||
MultiRootLocalFolderBackend,
|
||||
)
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
|
||||
_DOCUMENTS_MOUNT = "documents"
|
||||
|
||||
|
||||
def thread_working_copy_id(thread_id: object | None) -> str:
|
||||
"""The one place the thread → working-copy-id convention lives.
|
||||
|
||||
The file-op backend (here) and the end-of-turn commit middleware must
|
||||
resolve the same copy from the same thread: langgraph serializes turns per
|
||||
thread, and the middleware commits + discards the copy at end of turn.
|
||||
|
||||
Scoped to the turn, not the actor: subagents append ``::task:{id}`` per
|
||||
nesting level, so they resolve the root segment and share the parent's copy
|
||||
— the only one the commit reads, and one revision per turn.
|
||||
|
||||
ponytail: a copy left by a crashed turn is reused (and committed) by the
|
||||
thread's next turn — recovery semantics; abandoned threads are janitored.
|
||||
"""
|
||||
if thread_id is None:
|
||||
return "thread-adhoc"
|
||||
root = str(thread_id).split("::", 1)[0]
|
||||
# A parentless subagent's id is a bare ``task:{id}``, naming no turn.
|
||||
if not root or root.startswith("task:"):
|
||||
return "thread-adhoc"
|
||||
return f"thread-{root}"
|
||||
|
||||
|
||||
class GitTreeBackend:
|
||||
"""Serve ``/documents/...`` from the turn's private working copy.
|
||||
|
||||
A thin mount over the knowledge store's working copy: opened lazily on the
|
||||
first operation, plain file ops for the rest of the turn. No staging, no
|
||||
state overlay; the end-of-turn commit reads the copy's diff.
|
||||
"""
|
||||
|
||||
def __init__(self, workspace_id: int | str, runtime: ToolRuntime) -> None:
|
||||
self.workspace_id = workspace_id
|
||||
self._runtime = runtime
|
||||
self._mounted: MultiRootLocalFolderBackend | None = None
|
||||
|
||||
async def _backend(self) -> MultiRootLocalFolderBackend:
|
||||
if self._mounted is None:
|
||||
configurable = (self._runtime.config or {}).get("configurable") or {}
|
||||
copy_id = thread_working_copy_id(configurable.get("thread_id"))
|
||||
store = KnowledgeStore.for_workspace(self.workspace_id)
|
||||
copy = await store.open_working_copy(copy_id)
|
||||
# Mount the copy's documents/ subtree, not its root: the repo keeps
|
||||
# the documents/ prefix (C1), so agent writes must land under it —
|
||||
# the same paths the editor recorder and the migration seeder use.
|
||||
documents_root = copy.path / _DOCUMENTS_MOUNT
|
||||
documents_root.mkdir(exist_ok=True)
|
||||
self._mounted = MultiRootLocalFolderBackend(
|
||||
((_DOCUMENTS_MOUNT, str(documents_root)),)
|
||||
)
|
||||
return self._mounted
|
||||
|
||||
async def als_info(self, path: str) -> list[FileInfo]:
|
||||
return await (await self._backend()).als_info(path)
|
||||
|
||||
async def aread(self, file_path: str, offset: int = 0, limit: int = 2000) -> str:
|
||||
return await (await self._backend()).aread(file_path, offset, limit)
|
||||
|
||||
async def aread_raw(self, file_path: str) -> str:
|
||||
return await (await self._backend()).aread_raw(file_path)
|
||||
|
||||
async def awrite(self, file_path: str, content: str) -> WriteResult:
|
||||
return await (await self._backend()).awrite(file_path, content)
|
||||
|
||||
async def aedit(
|
||||
self,
|
||||
file_path: str,
|
||||
old_string: str,
|
||||
new_string: str,
|
||||
replace_all: bool = False,
|
||||
) -> EditResult:
|
||||
return await (await self._backend()).aedit(
|
||||
file_path, old_string, new_string, replace_all
|
||||
)
|
||||
|
||||
async def aglob_info(self, pattern: str, path: str = "/") -> list[FileInfo]:
|
||||
return await (await self._backend()).aglob_info(pattern, path)
|
||||
|
||||
async def agrep_raw(
|
||||
self,
|
||||
pattern: str,
|
||||
path: str | None = None,
|
||||
glob: str | None = None,
|
||||
) -> list[GrepMatch] | str:
|
||||
return await (await self._backend()).agrep_raw(pattern, path, glob)
|
||||
|
||||
async def alist_tree(
|
||||
self,
|
||||
path: str = "/",
|
||||
*,
|
||||
max_depth: int | None = 8,
|
||||
page_size: int = 500,
|
||||
include_files: bool = True,
|
||||
include_dirs: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
return await (await self._backend()).alist_tree(
|
||||
path,
|
||||
max_depth=max_depth,
|
||||
page_size=page_size,
|
||||
include_files=include_files,
|
||||
include_dirs=include_dirs,
|
||||
)
|
||||
|
||||
async def amove(
|
||||
self,
|
||||
source_path: str,
|
||||
destination_path: str,
|
||||
overwrite: bool = False,
|
||||
) -> WriteResult:
|
||||
return await (await self._backend()).amove(
|
||||
source_path, destination_path, overwrite
|
||||
)
|
||||
|
||||
async def adelete_file(self, file_path: str) -> WriteResult:
|
||||
return await (await self._backend()).adelete_file(file_path)
|
||||
|
||||
async def amkdir(
|
||||
self,
|
||||
dir_path: str,
|
||||
parents: bool = True,
|
||||
exist_ok: bool = True,
|
||||
) -> WriteResult:
|
||||
return await (await self._backend()).amkdir(
|
||||
dir_path, parents=parents, exist_ok=exist_ok
|
||||
)
|
||||
|
||||
async def armdir(self, dir_path: str) -> WriteResult:
|
||||
return await (await self._backend()).armdir(dir_path)
|
||||
|
||||
async def aupload_files(
|
||||
self, files: list[tuple[str, bytes]]
|
||||
) -> list[FileUploadResponse]:
|
||||
return await (await self._backend()).aupload_files(files)
|
||||
|
||||
async def adownload_files(self, paths: list[str]) -> list[FileDownloadResponse]:
|
||||
return await (await self._backend()).adownload_files(paths)
|
||||
|
|
@ -392,33 +392,6 @@ class LocalFolderBackend:
|
|||
async def adelete_file(self, file_path: str) -> WriteResult:
|
||||
return await asyncio.to_thread(self.delete_file, file_path)
|
||||
|
||||
def mkdir(
|
||||
self,
|
||||
dir_path: str,
|
||||
parents: bool = True,
|
||||
exist_ok: bool = True,
|
||||
) -> WriteResult:
|
||||
"""Create a directory under root so subsequent writes into it succeed."""
|
||||
try:
|
||||
path = self._resolve_virtual(dir_path)
|
||||
except ValueError:
|
||||
return WriteResult(error=f"Error: Invalid path '{dir_path}'")
|
||||
try:
|
||||
path.mkdir(parents=parents, exist_ok=exist_ok)
|
||||
except OSError as exc:
|
||||
return WriteResult(error=f"Error: failed to mkdir '{dir_path}': {exc}")
|
||||
return WriteResult(path=dir_path, files_update=None)
|
||||
|
||||
async def amkdir(
|
||||
self,
|
||||
dir_path: str,
|
||||
parents: bool = True,
|
||||
exist_ok: bool = True,
|
||||
) -> WriteResult:
|
||||
return await asyncio.to_thread(
|
||||
self.mkdir, dir_path, parents=parents, exist_ok=exist_ok
|
||||
)
|
||||
|
||||
def rmdir(self, dir_path: str) -> WriteResult:
|
||||
"""Hard-delete an empty directory under root.
|
||||
|
||||
|
|
|
|||
|
|
@ -300,36 +300,6 @@ class MultiRootLocalFolderBackend:
|
|||
async def adelete_file(self, file_path: str) -> WriteResult:
|
||||
return await asyncio.to_thread(self.delete_file, file_path)
|
||||
|
||||
def mkdir(
|
||||
self,
|
||||
dir_path: str,
|
||||
parents: bool = True,
|
||||
exist_ok: bool = True,
|
||||
) -> WriteResult:
|
||||
try:
|
||||
mount, local_path = self._split_mount_path(dir_path)
|
||||
except ValueError as exc:
|
||||
return WriteResult(error=f"Error: {exc}")
|
||||
if local_path == "/":
|
||||
# The mount root always exists.
|
||||
return WriteResult(path=dir_path, files_update=None)
|
||||
result = self._mount_to_backend[mount].mkdir(
|
||||
local_path, parents=parents, exist_ok=exist_ok
|
||||
)
|
||||
if result.path:
|
||||
result.path = self._prefix_mount_path(mount, result.path)
|
||||
return result
|
||||
|
||||
async def amkdir(
|
||||
self,
|
||||
dir_path: str,
|
||||
parents: bool = True,
|
||||
exist_ok: bool = True,
|
||||
) -> WriteResult:
|
||||
return await asyncio.to_thread(
|
||||
self.mkdir, dir_path, parents=parents, exist_ok=exist_ok
|
||||
)
|
||||
|
||||
def rmdir(self, dir_path: str) -> WriteResult:
|
||||
try:
|
||||
mount, local_path = self._split_mount_path(dir_path)
|
||||
|
|
|
|||
|
|
@ -13,9 +13,6 @@ from app.agents.chat.multi_agent_chat.shared.filesystem_selection import (
|
|||
FilesystemMode,
|
||||
FilesystemSelection,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.git_tree import (
|
||||
GitTreeBackend,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.kb_postgres import (
|
||||
KBPostgresBackend,
|
||||
)
|
||||
|
|
@ -35,20 +32,15 @@ def build_backend_resolver(
|
|||
selection: FilesystemSelection,
|
||||
*,
|
||||
workspace_id: int | None = None,
|
||||
knowledge_store_enabled: bool = False,
|
||||
) -> Callable[[ToolRuntime], BackendProtocol]:
|
||||
"""Create deepagents backend resolver for the selected filesystem mode.
|
||||
|
||||
In cloud mode the resolver returns a fresh :class:`KBPostgresBackend`
|
||||
bound to the current ``runtime`` so the backend can read staging state
|
||||
(``staged_dirs``, ``pending_moves``, ``files`` cache, ``kb_anon_doc``)
|
||||
for each tool call — or, when ``knowledge_store_enabled``, a
|
||||
:class:`GitTreeBackend` serving the turn's private working copy. The
|
||||
caller resolves that flag once per turn (global switch AND workspace
|
||||
flip), so a turn keeps one backend for its whole lifetime.
|
||||
When no ``workspace_id`` is provided, the resolver falls back to
|
||||
:class:`StateBackend` (used by sub-agents and tests that don't need
|
||||
DB-backed reads).
|
||||
for each tool call. When no ``workspace_id``
|
||||
is provided, the resolver falls back to :class:`StateBackend` (used by
|
||||
sub-agents and tests that don't need DB-backed reads).
|
||||
|
||||
Desktop-local mode unchanged.
|
||||
"""
|
||||
|
|
@ -64,12 +56,6 @@ def build_backend_resolver(
|
|||
return _resolve_local
|
||||
|
||||
if workspace_id is not None:
|
||||
if knowledge_store_enabled:
|
||||
|
||||
def _resolve_git_tree(runtime: ToolRuntime) -> BackendProtocol:
|
||||
return GitTreeBackend(workspace_id, runtime)
|
||||
|
||||
return _resolve_git_tree
|
||||
|
||||
def _resolve_kb(runtime: ToolRuntime) -> BackendProtocol:
|
||||
return KBPostgresBackend(workspace_id, runtime)
|
||||
|
|
|
|||
|
|
@ -17,7 +17,6 @@ def build_filesystem_mw(
|
|||
user_id: str | None,
|
||||
thread_id: int | None,
|
||||
read_only: bool = False,
|
||||
knowledge_store_enabled: bool = False,
|
||||
) -> SurfSenseFilesystemMiddleware:
|
||||
return SurfSenseFilesystemMiddleware(
|
||||
backend=backend_resolver,
|
||||
|
|
@ -26,5 +25,4 @@ def build_filesystem_mw(
|
|||
created_by_id=user_id,
|
||||
thread_id=thread_id,
|
||||
read_only=read_only,
|
||||
knowledge_store_enabled=knowledge_store_enabled,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -54,12 +54,8 @@ class SurfSenseFilesystemMiddleware(FilesystemMiddleware):
|
|||
thread_id: int | str | None = None,
|
||||
tool_token_limit_before_evict: int | None = 20000,
|
||||
read_only: bool = False,
|
||||
knowledge_store_enabled: bool = False,
|
||||
) -> None:
|
||||
self._filesystem_mode = filesystem_mode
|
||||
# Only ``read_file`` reads this, to describe honestly whether its reads
|
||||
# carry a citation envelope; the backend choice itself is the resolver's.
|
||||
self._knowledge_store_enabled = knowledge_store_enabled
|
||||
self._workspace_id = workspace_id
|
||||
self._created_by_id = created_by_id
|
||||
self._thread_id = thread_id
|
||||
|
|
|
|||
|
|
@ -11,9 +11,6 @@ from langchain_core.messages import ToolMessage
|
|||
from langchain_core.tools import BaseTool, StructuredTool
|
||||
from langgraph.types import Command
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.git_tree import (
|
||||
GitTreeBackend,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.kb_postgres import (
|
||||
KBPostgresBackend,
|
||||
)
|
||||
|
|
@ -98,8 +95,7 @@ def create_edit_file_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
|||
)
|
||||
],
|
||||
}
|
||||
# The git-tree backend already edited the working copy; no staging.
|
||||
if is_cloud(mw._filesystem_mode) and not isinstance(backend, GitTreeBackend):
|
||||
if is_cloud(mw._filesystem_mode):
|
||||
update["dirty_paths"] = [path]
|
||||
update["dirty_path_tool_calls"] = {path: runtime.tool_call_id}
|
||||
if doc_id_to_attach is not None:
|
||||
|
|
|
|||
|
|
@ -11,9 +11,6 @@ from langchain_core.messages import ToolMessage
|
|||
from langchain_core.tools import BaseTool, StructuredTool
|
||||
from langgraph.types import Command
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.git_tree import (
|
||||
GitTreeBackend,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.state.filesystem_state import (
|
||||
SurfSenseFilesystemState,
|
||||
)
|
||||
|
|
@ -41,9 +38,7 @@ def create_mkdir_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
|||
except ValueError as exc:
|
||||
return f"Error: {exc}"
|
||||
|
||||
backend = mw._get_backend(runtime)
|
||||
# The git-tree backend needs no staging: directories materialize with writes.
|
||||
if is_cloud(mw._filesystem_mode) and not isinstance(backend, GitTreeBackend):
|
||||
if is_cloud(mw._filesystem_mode):
|
||||
if not (
|
||||
validated.startswith(DOCUMENTS_ROOT + "/")
|
||||
or validated == DOCUMENTS_ROOT
|
||||
|
|
@ -70,6 +65,7 @@ def create_mkdir_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
|||
}
|
||||
)
|
||||
|
||||
backend = mw._get_backend(runtime)
|
||||
local_method = getattr(backend, "amkdir", None) or getattr(
|
||||
backend, "mkdir", None
|
||||
)
|
||||
|
|
@ -77,16 +73,13 @@ def create_mkdir_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
|||
try:
|
||||
res: Any = local_method(validated, parents=True, exist_ok=True)
|
||||
if asyncio.iscoroutine(res):
|
||||
res = await res
|
||||
await res
|
||||
except TypeError:
|
||||
res = local_method(validated)
|
||||
if asyncio.iscoroutine(res):
|
||||
res = await res
|
||||
await res
|
||||
except Exception as exc: # pragma: no cover
|
||||
return f"Error: {exc}"
|
||||
error = getattr(res, "error", None)
|
||||
if error:
|
||||
return error
|
||||
return f"Created directory {validated}"
|
||||
|
||||
def sync_mkdir(
|
||||
|
|
|
|||
|
|
@ -11,9 +11,6 @@ from langchain_core.messages import ToolMessage
|
|||
from langchain_core.tools import BaseTool, StructuredTool
|
||||
from langgraph.types import Command
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.git_tree import (
|
||||
GitTreeBackend,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.state.filesystem_state import (
|
||||
SurfSenseFilesystemState,
|
||||
)
|
||||
|
|
@ -52,9 +49,7 @@ def create_move_file_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
|||
except ValueError as exc:
|
||||
return f"Error: {exc}"
|
||||
|
||||
backend = mw._get_backend(runtime)
|
||||
# The git-tree backend moves directly on the working copy; no staging.
|
||||
if is_cloud(mw._filesystem_mode) and not isinstance(backend, GitTreeBackend):
|
||||
if is_cloud(mw._filesystem_mode):
|
||||
return await cloud_move_file(
|
||||
mw,
|
||||
runtime,
|
||||
|
|
@ -63,6 +58,7 @@ def create_move_file_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
|||
overwrite=overwrite,
|
||||
)
|
||||
|
||||
backend = mw._get_backend(runtime)
|
||||
res: WriteResult = await backend.amove(
|
||||
validated_source, validated_dest, overwrite=overwrite
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,52 +1,22 @@
|
|||
"""Description strings for ``read_file``, split by whether reads are citable.
|
||||
|
||||
The split is **not** cloud vs desktop like its sibling tools: it is whether the
|
||||
read returns a citation envelope. Only cloud-on-Postgres does, because only
|
||||
``KBPostgresBackend`` renders documents through ``render_full_document``; the
|
||||
desktop mounts and the git-native working copy both return raw file text. A
|
||||
single description promising ``[n]``-labelled passages therefore misinstructs
|
||||
two of the three modes, and the failure is worse than a missing citation: told
|
||||
to cite "the same ``[n]`` you would use from ``search_knowledge_base``" while
|
||||
seeing no labels, a model can attach a search result's ordinal to text it read
|
||||
from a file, producing a confident citation pointing at the wrong source.
|
||||
"""
|
||||
"""Description string for ``read_file`` (mode-agnostic)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.filesystem_selection import FilesystemMode
|
||||
|
||||
_USAGE = """Reads a file from the filesystem.
|
||||
_DESCRIPTION = """Reads a file from the filesystem.
|
||||
|
||||
Usage:
|
||||
- By default, reads up to 100 lines from the beginning.
|
||||
- Use `offset` and `limit` for pagination when files are large.
|
||||
- Results include line numbers.
|
||||
"""
|
||||
|
||||
_ENVELOPE_DESCRIPTION = (
|
||||
_USAGE
|
||||
+ """- A knowledge-base document is returned as a `<document … view="full">` block:
|
||||
- A knowledge-base document is returned as a `<document … view="full">` block:
|
||||
the whole source, with each passage labelled `[n]`. `view="full"` means you are
|
||||
seeing the complete document, not an excerpt.
|
||||
- Cite a passage by writing its `[n]` after the statement it supports — the same
|
||||
`[n]` you would use for that passage from `search_knowledge_base`.
|
||||
"""
|
||||
)
|
||||
|
||||
_RAW_DESCRIPTION = (
|
||||
_USAGE
|
||||
+ """- The result is the file's own text, with no `[n]` labels, so there is nothing
|
||||
here to cite. Never attach an `[n]` to a statement taken from a read: the
|
||||
ordinals you have seen belong to `search_knowledge_base` excerpts, and reusing
|
||||
one here would credit the wrong source. Cite the same content from
|
||||
`search_knowledge_base`, which does label what it returns, or state it without
|
||||
a citation.
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def select_description(mode: FilesystemMode, *, git_native: bool = False) -> str:
|
||||
"""Pick the description matching what this mode's reads actually return."""
|
||||
if mode == FilesystemMode.CLOUD and not git_native:
|
||||
return _ENVELOPE_DESCRIPTION
|
||||
return _RAW_DESCRIPTION
|
||||
def select_description(mode: FilesystemMode) -> str:
|
||||
return _DESCRIPTION
|
||||
|
|
|
|||
|
|
@ -32,9 +32,7 @@ if TYPE_CHECKING:
|
|||
|
||||
|
||||
def create_read_file_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
||||
description = select_description(
|
||||
mw._filesystem_mode, git_native=mw._knowledge_store_enabled
|
||||
)
|
||||
description = select_description(mw._filesystem_mode)
|
||||
|
||||
async def async_read_file(
|
||||
file_path: Annotated[
|
||||
|
|
|
|||
|
|
@ -9,9 +9,6 @@ from langchain.tools import ToolRuntime
|
|||
from langchain_core.tools import BaseTool, StructuredTool
|
||||
from langgraph.types import Command
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.git_tree import (
|
||||
GitTreeBackend,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.state.filesystem_state import (
|
||||
SurfSenseFilesystemState,
|
||||
)
|
||||
|
|
@ -45,10 +42,7 @@ def create_rm_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
|||
except ValueError as exc:
|
||||
return f"Error: {exc}"
|
||||
|
||||
# The git-tree backend deletes directly on the working copy; no staging.
|
||||
if is_cloud(mw._filesystem_mode) and not isinstance(
|
||||
mw._get_backend(runtime), GitTreeBackend
|
||||
):
|
||||
if is_cloud(mw._filesystem_mode):
|
||||
return await cloud_rm(mw, runtime, validated)
|
||||
return await desktop_rm(mw, runtime, validated)
|
||||
|
||||
|
|
|
|||
|
|
@ -9,9 +9,6 @@ from langchain.tools import ToolRuntime
|
|||
from langchain_core.tools import BaseTool, StructuredTool
|
||||
from langgraph.types import Command
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.git_tree import (
|
||||
GitTreeBackend,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.state.filesystem_state import (
|
||||
SurfSenseFilesystemState,
|
||||
)
|
||||
|
|
@ -45,10 +42,7 @@ def create_rmdir_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
|||
except ValueError as exc:
|
||||
return f"Error: {exc}"
|
||||
|
||||
# The git-tree backend removes directly on the working copy; no staging.
|
||||
if is_cloud(mw._filesystem_mode) and not isinstance(
|
||||
mw._get_backend(runtime), GitTreeBackend
|
||||
):
|
||||
if is_cloud(mw._filesystem_mode):
|
||||
return await cloud_rmdir(mw, runtime, validated)
|
||||
return await desktop_rmdir(mw, runtime, validated)
|
||||
|
||||
|
|
|
|||
|
|
@ -11,9 +11,6 @@ from langchain_core.messages import ToolMessage
|
|||
from langchain_core.tools import BaseTool, StructuredTool
|
||||
from langgraph.types import Command
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.git_tree import (
|
||||
GitTreeBackend,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.state.filesystem_state import (
|
||||
SurfSenseFilesystemState,
|
||||
)
|
||||
|
|
@ -65,8 +62,7 @@ def create_write_file_tool(mw: SurfSenseFilesystemMiddleware) -> BaseTool:
|
|||
)
|
||||
],
|
||||
}
|
||||
# The git-tree backend already wrote to the working copy; no staging.
|
||||
if is_cloud(mw._filesystem_mode) and not isinstance(backend, GitTreeBackend):
|
||||
if is_cloud(mw._filesystem_mode):
|
||||
update["dirty_paths"] = [path]
|
||||
update["dirty_path_tool_calls"] = {path: runtime.tool_call_id}
|
||||
return Command(update=update)
|
||||
|
|
|
|||
|
|
@ -119,7 +119,6 @@ def build_kb_middleware(
|
|||
user_id=dependencies.get("user_id"),
|
||||
thread_id=dependencies.get("thread_id"),
|
||||
read_only=read_only,
|
||||
knowledge_store_enabled=bool(dependencies.get("knowledge_store_enabled")),
|
||||
)
|
||||
_t_fs = _perf_time.perf_counter() - _t0
|
||||
|
||||
|
|
|
|||
|
|
@ -39,7 +39,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||
from app.agents.chat.runtime.path_resolver import (
|
||||
DOCUMENTS_ROOT,
|
||||
build_path_index,
|
||||
virtual_path_of,
|
||||
doc_to_virtual_path,
|
||||
)
|
||||
from app.db import Document, Folder
|
||||
from app.schemas.new_chat import MentionedDocumentInfo
|
||||
|
|
@ -189,8 +189,7 @@ async def resolve_mentions(
|
|||
)
|
||||
continue
|
||||
title = chip_titles_by_id.get(("doc", doc_id), str(row.title or ""))
|
||||
path = virtual_path_of(
|
||||
metadata=row.document_metadata,
|
||||
path = doc_to_virtual_path(
|
||||
doc_id=row.id,
|
||||
title=str(row.title or "untitled"),
|
||||
folder_id=row.folder_id,
|
||||
|
|
|
|||
|
|
@ -15,9 +15,7 @@ commits.
|
|||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
|
@ -28,44 +26,6 @@ from app.utils.document_converters import generate_unique_identifier_hash
|
|||
DOCUMENTS_ROOT = "/documents"
|
||||
"""Root virtual folder for all KB documents."""
|
||||
|
||||
PATH_MARKER = "virtual_path"
|
||||
"""``document_metadata`` key holding the virtual path a row's content lives at.
|
||||
|
||||
Written by the store indexer on every converged row, and by the revision
|
||||
recorder when a recorded save lands at a new path. Its presence marks a row as
|
||||
path-addressed (indexer-owned for pruning); its value is the file a retitle
|
||||
must drop from the tree.
|
||||
"""
|
||||
|
||||
|
||||
def to_store_path(virtual_path: str) -> str:
|
||||
"""Convert an agent-facing ``/documents/...`` path to its git-repo path.
|
||||
|
||||
The repo tree keeps the ``documents/`` root (C1 as shipped — the top level
|
||||
stays free for future sibling roots like ``.cache/``), so this only drops
|
||||
the leading slash. It still raises on a foreign namespace: a silent
|
||||
mismatch here forks one document into two identities on either side of the
|
||||
git↔Postgres boundary.
|
||||
"""
|
||||
if virtual_path != DOCUMENTS_ROOT and not virtual_path.startswith(
|
||||
f"{DOCUMENTS_ROOT}/"
|
||||
):
|
||||
msg = f"Not a {DOCUMENTS_ROOT} path: {virtual_path!r}"
|
||||
raise ValueError(msg)
|
||||
return virtual_path.lstrip("/")
|
||||
|
||||
|
||||
def to_virtual_path(store_path: str) -> str:
|
||||
"""Convert a git-repo path back to its agent-facing ``/documents/...`` path.
|
||||
|
||||
Inverse of :func:`to_store_path`. Every identity derived from the store — the
|
||||
``unique_identifier_hash`` and the ``PATH_MARKER`` metadata — is keyed on the
|
||||
virtual path, so callers convert once on the way in and stay in one
|
||||
namespace from there.
|
||||
"""
|
||||
return f"/{store_path.strip('/')}"
|
||||
|
||||
|
||||
_INVALID_FILENAME_CHARS = re.compile(r"[\\/:*?\"<>|]+")
|
||||
_WHITESPACE_RUN = re.compile(r"\s+")
|
||||
|
||||
|
|
@ -226,36 +186,6 @@ def doc_to_virtual_path(
|
|||
return path
|
||||
|
||||
|
||||
def virtual_path_of(
|
||||
*,
|
||||
metadata: Mapping[str, Any] | None,
|
||||
doc_id: int | None,
|
||||
title: str,
|
||||
folder_id: int | None,
|
||||
index: PathIndex,
|
||||
) -> str:
|
||||
"""Where a row's content lives, per its :data:`PATH_MARKER`.
|
||||
|
||||
Two writers name files: the seeder and the revision recorder derive a name
|
||||
from the title, while the agent's ``write_file`` commits whatever name the
|
||||
model chose. Deriving is therefore a guess about anything the agent authored,
|
||||
and the marker is the only record that survives the disagreement. Rows with
|
||||
no marker fall back to derivation — which is the name the seeder gave them.
|
||||
|
||||
Ask :func:`doc_to_virtual_path` instead when the question is where a document
|
||||
*should* live: a retitle needs the title's answer to know what to move.
|
||||
"""
|
||||
recorded = (metadata or {}).get(PATH_MARKER)
|
||||
if isinstance(recorded, str) and recorded.startswith(f"{DOCUMENTS_ROOT}/"):
|
||||
# Claim the slot, or a later derived path could be handed the same one.
|
||||
if doc_id is not None:
|
||||
index.occupants[recorded] = doc_id
|
||||
return recorded
|
||||
return doc_to_virtual_path(
|
||||
doc_id=doc_id, title=title, folder_id=folder_id, index=index
|
||||
)
|
||||
|
||||
|
||||
async def virtual_path_to_doc(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
|
|
@ -409,7 +339,6 @@ def parse_documents_path(virtual_path: str) -> tuple[list[str], str]:
|
|||
|
||||
__all__ = [
|
||||
"DOCUMENTS_ROOT",
|
||||
"PATH_MARKER",
|
||||
"PathIndex",
|
||||
"build_path_index",
|
||||
"doc_to_virtual_path",
|
||||
|
|
@ -417,8 +346,5 @@ __all__ = [
|
|||
"parse_documents_path",
|
||||
"safe_filename",
|
||||
"safe_folder_segment",
|
||||
"to_store_path",
|
||||
"to_virtual_path",
|
||||
"virtual_path_of",
|
||||
"virtual_path_to_doc",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ from __future__ import annotations
|
|||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.agents.chat.runtime.path_resolver import PathIndex, virtual_path_of
|
||||
from app.agents.chat.runtime.path_resolver import PathIndex, doc_to_virtual_path
|
||||
from app.db import Document
|
||||
|
||||
from ..models import DocumentReference
|
||||
|
|
@ -44,8 +44,7 @@ async def resolve_document_references(
|
|||
DocumentReference(
|
||||
entity_id=document.id,
|
||||
label=title,
|
||||
path=virtual_path_of(
|
||||
metadata=document.document_metadata,
|
||||
path=doc_to_virtual_path(
|
||||
doc_id=document.id,
|
||||
title=title,
|
||||
folder_id=document.folder_id,
|
||||
|
|
|
|||
|
|
@ -202,9 +202,6 @@ celery_app = Celery(
|
|||
"app.tasks.celery_tasks.stale_notification_cleanup_task",
|
||||
"app.tasks.celery_tasks.stripe_reconciliation_task",
|
||||
"app.tasks.celery_tasks.refresh_token_cleanup_task",
|
||||
"app.tasks.celery_tasks.knowledge_store.janitor_task",
|
||||
"app.tasks.celery_tasks.knowledge_store.index_tasks",
|
||||
"app.tasks.celery_tasks.knowledge_store.drift_monitor_task",
|
||||
"app.tasks.celery_tasks.auto_reload_task",
|
||||
"app.tasks.celery_tasks.gateway_tasks",
|
||||
"app.etl_pipeline.cache.eviction.task",
|
||||
|
|
@ -263,9 +260,6 @@ celery_app.conf.update(
|
|||
"index_bookstack_pages": {"queue": CONNECTORS_QUEUE},
|
||||
"index_composio_connector": {"queue": CONNECTORS_QUEUE},
|
||||
"index_obsidian_attachment": {"queue": CONNECTORS_QUEUE},
|
||||
# A whole-workspace rebuild embeds every document; the per-save index
|
||||
# task stays on the fast queue because search freshness is user-facing.
|
||||
"reindex_knowledge_store": {"queue": CONNECTORS_QUEUE},
|
||||
# Everything else (document processing, podcasts, reindexing,
|
||||
# schedule checker, cleanup) stays on the default fast queue.
|
||||
"gateway.reconcile_inbox": {"queue": f"{CELERY_TASK_DEFAULT_QUEUE}.gateway"},
|
||||
|
|
@ -342,29 +336,6 @@ celery_app.conf.beat_schedule = {
|
|||
"schedule": crontab(hour="4", minute="30"),
|
||||
"options": {"expires": 600},
|
||||
},
|
||||
# Prune knowledge-store working copies abandoned by crashed threads.
|
||||
"prune-knowledge-store-working-copies": {
|
||||
"task": "prune_knowledge_store_working_copies",
|
||||
"schedule": crontab(hour="4", minute="45"),
|
||||
"options": {"expires": 600},
|
||||
},
|
||||
# Re-drive flipped workspaces whose chunk index trails their store. Hourly,
|
||||
# not daily: it is the only recovery for an index task lost to a broker or
|
||||
# worker failure, and it backfills a workspace flipped before its first
|
||||
# index run.
|
||||
"reindex-drifted-workspaces": {
|
||||
"task": "reindex_drifted_workspaces",
|
||||
"schedule": crontab(minute="20"),
|
||||
"options": {"expires": 600},
|
||||
},
|
||||
# Parity-check flipped workspaces against git by content address. Covers the
|
||||
# half the hourly sweep cannot see (its predicate is git-vs-git), and enqueues
|
||||
# a whole-tree converge for what it finds, capped per run.
|
||||
"check-knowledge-store-drift": {
|
||||
"task": "check_knowledge_store_drift",
|
||||
"schedule": crontab(hour="5", minute="15"),
|
||||
"options": {"expires": 600},
|
||||
},
|
||||
# Fire due automation schedule triggers (Beat entry owned by the schedule
|
||||
# trigger; see app.automations.triggers.builtin.schedule.source).
|
||||
**SCHEDULE_BEAT_SCHEDULE,
|
||||
|
|
|
|||
|
|
@ -538,16 +538,6 @@ class Config:
|
|||
"FILE_STORAGE_LOCAL_PATH", str(BASE_DIR / ".local_object_store")
|
||||
)
|
||||
|
||||
# Knowledge store (Git-native KB; off by default). Nested under the shared
|
||||
# file-storage volume so every process sees the same history.
|
||||
KNOWLEDGE_STORE_ENABLED = (
|
||||
os.getenv("KNOWLEDGE_STORE_ENABLED", "FALSE").upper() == "TRUE"
|
||||
)
|
||||
KNOWLEDGE_STORE_ROOT = os.getenv(
|
||||
"KNOWLEDGE_STORE_ROOT",
|
||||
os.path.join(FILE_STORAGE_LOCAL_PATH, "knowledge_store"),
|
||||
)
|
||||
|
||||
# Daytona sandbox (code execution / filesystem sandbox)
|
||||
DAYTONA_SANDBOX_ENABLED = (
|
||||
os.getenv("DAYTONA_SANDBOX_ENABLED", "FALSE").upper() == "TRUE"
|
||||
|
|
|
|||
|
|
@ -1475,13 +1475,6 @@ class Chunk(BaseModel, TimestampMixin):
|
|||
# building a position index on the large chunks table is not worth it.
|
||||
position = Column(Integer, nullable=False, server_default="0")
|
||||
|
||||
# 1-based inclusive line range this chunk was cut from, in the document's
|
||||
# source markdown. Sole consumer is rendering true line numbers on search
|
||||
# excerpts; never a stored reference, so a rebuild cannot strand it.
|
||||
# NULL on rows written before spans existed.
|
||||
start_line = Column(Integer, nullable=True)
|
||||
end_line = Column(Integer, nullable=True)
|
||||
|
||||
document_id = Column(
|
||||
Integer,
|
||||
ForeignKey("documents.id", ondelete="CASCADE"),
|
||||
|
|
@ -1744,18 +1737,6 @@ class Workspace(BaseModel, TimestampMixin):
|
|||
# verdict into first-run vs. recovery.
|
||||
llm_setup_completed_at = Column(TIMESTAMP(timezone=True), nullable=True)
|
||||
|
||||
# Progressive git-native flip: this workspace uses the git-backed knowledge
|
||||
# store only when this AND the global KNOWLEDGE_STORE_ENABLED are true.
|
||||
# Flipped per workspace after its migration seed passes parity.
|
||||
knowledge_store_enabled = Column(
|
||||
Boolean, nullable=False, default=False, server_default="false"
|
||||
)
|
||||
|
||||
# Revision the derived index (chunks + embeddings) was last built from.
|
||||
# NULL means never indexed; ``!= store.get_current_revision()`` is the drift
|
||||
# predicate the sweep uses to re-drive a workspace.
|
||||
last_indexed_revision = Column(String(64), nullable=True)
|
||||
|
||||
user_id = Column(
|
||||
UUID(as_uuid=True), ForeignKey("user.id", ondelete="CASCADE"), nullable=False
|
||||
)
|
||||
|
|
|
|||
|
|
@ -10,7 +10,6 @@ from __future__ import annotations
|
|||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
|
@ -19,12 +18,7 @@ from app.indexing_pipeline.cache.eligibility import is_embedding_cacheable
|
|||
from app.indexing_pipeline.cache.schemas import CachedChunk, EmbeddingKey, EmbeddingSet
|
||||
from app.indexing_pipeline.cache.service import EmbeddingCacheService
|
||||
from app.indexing_pipeline.cache.settings import load_embedding_cache_settings
|
||||
from app.indexing_pipeline.document_chunker import (
|
||||
LineChunk,
|
||||
attach_line_spans,
|
||||
chunk_text,
|
||||
chunk_text_hybrid,
|
||||
)
|
||||
from app.indexing_pipeline.document_chunker import chunk_text, chunk_text_hybrid
|
||||
from app.indexing_pipeline.document_embedder import embed_texts
|
||||
from app.observability import metrics
|
||||
|
||||
|
|
@ -33,27 +27,13 @@ logger = logging.getLogger(__name__)
|
|||
ChunkPair = tuple[str, np.ndarray]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class EmbeddedChunk:
|
||||
"""One chunk ready to persist: its text, its line range, and its vector."""
|
||||
|
||||
text: str
|
||||
start_line: int
|
||||
end_line: int
|
||||
embedding: np.ndarray
|
||||
|
||||
|
||||
async def build_chunk_embeddings(
|
||||
markdown: str, *, use_code_chunker: bool
|
||||
) -> tuple[np.ndarray, list[EmbeddedChunk]]:
|
||||
"""Return the document-level vector and the ordered chunks to persist.
|
||||
) -> tuple[np.ndarray, list[ChunkPair]]:
|
||||
"""Return the document-level vector and ordered ``(chunk_text, vector)`` pairs.
|
||||
|
||||
Drop-in for the inline chunk+embed step; reuses prior output when the same
|
||||
markdown has already been embedded with the current model and chunker.
|
||||
|
||||
Line spans are attached here rather than cached: they are a pure function of
|
||||
the markdown and the chunk texts, both of which every caller already holds,
|
||||
so caching them would buy nothing and would invalidate every existing entry.
|
||||
"""
|
||||
settings = load_embedding_cache_settings()
|
||||
chunker_kind = "code" if use_code_chunker else "hybrid"
|
||||
|
|
@ -65,10 +45,7 @@ async def build_chunk_embeddings(
|
|||
embedding_dim=embedding_dim,
|
||||
)
|
||||
if not cacheable:
|
||||
summary_embedding, chunk_pairs = await _compute(
|
||||
markdown, use_code_chunker=use_code_chunker
|
||||
)
|
||||
return summary_embedding, _with_line_spans(markdown, chunk_pairs)
|
||||
return await _compute(markdown, use_code_chunker=use_code_chunker)
|
||||
|
||||
key = EmbeddingKey(
|
||||
markdown_sha256=_hash_text(markdown),
|
||||
|
|
@ -86,9 +63,7 @@ async def build_chunk_embeddings(
|
|||
outcome="hit",
|
||||
)
|
||||
logger.debug("Embedding cache hit for %s", key.markdown_sha256)
|
||||
return cached.summary_embedding, _with_line_spans(
|
||||
markdown, [(c.text, c.embedding) for c in cached.chunks]
|
||||
)
|
||||
return cached.summary_embedding, [(c.text, c.embedding) for c in cached.chunks]
|
||||
|
||||
metrics.record_embedding_cache_lookup(
|
||||
embedding_model=key.embedding_model, chunker_kind=chunker_kind, outcome="miss"
|
||||
|
|
@ -97,7 +72,7 @@ async def build_chunk_embeddings(
|
|||
markdown, use_code_chunker=use_code_chunker
|
||||
)
|
||||
await _remember(key, summary_embedding, chunk_pairs)
|
||||
return summary_embedding, _with_line_spans(markdown, chunk_pairs)
|
||||
return summary_embedding, chunk_pairs
|
||||
|
||||
|
||||
async def chunk_markdown(markdown: str, *, use_code_chunker: bool) -> list[str]:
|
||||
|
|
@ -108,14 +83,6 @@ async def chunk_markdown(markdown: str, *, use_code_chunker: bool) -> list[str]:
|
|||
return await asyncio.to_thread(chunk_text_hybrid, markdown)
|
||||
|
||||
|
||||
async def chunk_markdown_with_lines(
|
||||
markdown: str, *, use_code_chunker: bool
|
||||
) -> list[LineChunk]:
|
||||
"""Chunk markdown into ordered texts, each carrying its line range."""
|
||||
texts = await chunk_markdown(markdown, use_code_chunker=use_code_chunker)
|
||||
return attach_line_spans(markdown, texts)
|
||||
|
||||
|
||||
async def embed_batch(texts: list[str]) -> list[np.ndarray]:
|
||||
"""Embed texts in one batch off the event loop."""
|
||||
return await asyncio.to_thread(embed_texts, texts)
|
||||
|
|
@ -158,18 +125,5 @@ async def _remember(
|
|||
logger.warning("Embedding cache write failed; result not cached", exc_info=True)
|
||||
|
||||
|
||||
def _with_line_spans(markdown: str, pairs: list[ChunkPair]) -> list[EmbeddedChunk]:
|
||||
spans = attach_line_spans(markdown, [text for text, _ in pairs])
|
||||
return [
|
||||
EmbeddedChunk(
|
||||
text=span.text,
|
||||
start_line=span.start_line,
|
||||
end_line=span.end_line,
|
||||
embedding=embedding,
|
||||
)
|
||||
for (_, embedding), span in zip(pairs, spans, strict=True)
|
||||
]
|
||||
|
||||
|
||||
def _hash_text(text: str) -> str:
|
||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
|
|
|
|||
|
|
@ -13,89 +13,44 @@ from __future__ import annotations
|
|||
from collections import defaultdict, deque
|
||||
from dataclasses import dataclass
|
||||
|
||||
from app.indexing_pipeline.document_chunker import LineChunk
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ExistingChunk:
|
||||
id: int
|
||||
content: str
|
||||
position: int
|
||||
#: ``None`` on rows written before line spans existed.
|
||||
start_line: int | None = None
|
||||
end_line: int | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReusedChunk:
|
||||
"""A kept row whose position or line range needs writing back."""
|
||||
|
||||
id: int
|
||||
position: int
|
||||
start_line: int
|
||||
end_line: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PendingChunk:
|
||||
"""A new text that has to be embedded and inserted."""
|
||||
|
||||
position: int
|
||||
text: str
|
||||
start_line: int
|
||||
end_line: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ChunkPlan:
|
||||
"""The minimal set of writes that turns the stored chunks into the new ones.
|
||||
|
||||
``reused`` holds only kept rows that actually changed — position or line
|
||||
range; rows identical in both need no write at all. Kept-row count (for
|
||||
metrics) is ``len(existing) - len(to_delete)``.
|
||||
``reused`` holds only kept rows whose position actually changed; rows that
|
||||
match in place need no write at all. Kept-row count (for metrics) is
|
||||
``len(existing) - len(to_delete)``.
|
||||
"""
|
||||
|
||||
reused: list[ReusedChunk]
|
||||
to_embed: list[PendingChunk]
|
||||
reused: list[tuple[int, int]] # (existing_chunk_id, new_position)
|
||||
to_embed: list[tuple[int, str]] # (new_position, text)
|
||||
to_delete: list[int] # existing chunk ids
|
||||
|
||||
|
||||
def reconcile(existing: list[ExistingChunk], new_chunks: list[LineChunk]) -> ChunkPlan:
|
||||
def reconcile(existing: list[ExistingChunk], new_texts: list[str]) -> ChunkPlan:
|
||||
available: dict[str, deque[ExistingChunk]] = defaultdict(deque)
|
||||
for chunk in sorted(existing, key=lambda c: c.position):
|
||||
available[chunk.content].append(chunk)
|
||||
|
||||
reused: list[ReusedChunk] = []
|
||||
to_embed: list[PendingChunk] = []
|
||||
reused: list[tuple[int, int]] = []
|
||||
to_embed: list[tuple[int, str]] = []
|
||||
|
||||
for new_position, new_chunk in enumerate(new_chunks):
|
||||
matches = available.get(new_chunk.text)
|
||||
for new_position, text in enumerate(new_texts):
|
||||
matches = available.get(text)
|
||||
if matches:
|
||||
chunk = matches.popleft()
|
||||
# Unchanged text still moves when a paragraph is inserted above it:
|
||||
# same embedding, different lines. Both have to be written back.
|
||||
if (chunk.position, chunk.start_line, chunk.end_line) != (
|
||||
new_position,
|
||||
new_chunk.start_line,
|
||||
new_chunk.end_line,
|
||||
):
|
||||
reused.append(
|
||||
ReusedChunk(
|
||||
id=chunk.id,
|
||||
position=new_position,
|
||||
start_line=new_chunk.start_line,
|
||||
end_line=new_chunk.end_line,
|
||||
)
|
||||
)
|
||||
if chunk.position != new_position:
|
||||
reused.append((chunk.id, new_position))
|
||||
else:
|
||||
to_embed.append(
|
||||
PendingChunk(
|
||||
position=new_position,
|
||||
text=new_chunk.text,
|
||||
start_line=new_chunk.start_line,
|
||||
end_line=new_chunk.end_line,
|
||||
)
|
||||
)
|
||||
to_embed.append((new_position, text))
|
||||
|
||||
to_delete = [chunk.id for queue in available.values() for chunk in queue]
|
||||
return ChunkPlan(reused=reused, to_embed=to_embed, to_delete=to_delete)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,4 @@
|
|||
import re
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
|
||||
from app.config import config
|
||||
|
||||
|
|
@ -59,42 +57,3 @@ def chunk_text_hybrid(text: str) -> list[str]:
|
|||
chunks.extend(chunk_text(trailing))
|
||||
|
||||
return chunks
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LineChunk:
|
||||
"""A chunk text plus the 1-based inclusive line range it was cut from."""
|
||||
|
||||
text: str
|
||||
start_line: int
|
||||
end_line: int
|
||||
|
||||
|
||||
def attach_line_spans(text: str, chunks: Sequence[str]) -> list[LineChunk]:
|
||||
"""Locate ordered, non-overlapping ``chunks`` in ``text`` as line ranges.
|
||||
|
||||
Chunks arrive in document order, so a left-to-right cursor finds each one
|
||||
unambiguously even in a document that repeats a line — the ambiguity only
|
||||
exists for a whole-document search.
|
||||
|
||||
``ponytail:`` located by ordered search rather than by chunker-reported
|
||||
offsets, which leaves ``chunk_text``/``chunk_text_hybrid`` (and every test
|
||||
seam that patches them) alone, and keeps the hybrid chunker's ``.strip()``
|
||||
from needing offset bookkeeping. Ceiling: a chunker that emits overlapping
|
||||
windows or rewrites chunk text falls back to the cursor line; upgrade path
|
||||
is to thread the chunker's own ``start_index`` through instead.
|
||||
"""
|
||||
spans: list[LineChunk] = []
|
||||
cursor = 0
|
||||
cursor_line = 1
|
||||
for chunk in chunks:
|
||||
found = text.find(chunk, cursor)
|
||||
start = found if found >= 0 else cursor
|
||||
start_line = cursor_line + text.count("\n", cursor, start)
|
||||
end = start + len(chunk)
|
||||
# ``end - 1`` so a chunk ending on a newline does not claim the next line.
|
||||
end_line = start_line + text.count("\n", start, max(end - 1, start))
|
||||
spans.append(LineChunk(text=chunk, start_line=start_line, end_line=end_line))
|
||||
cursor = end
|
||||
cursor_line = start_line + text.count("\n", start, end)
|
||||
return spans
|
||||
|
|
|
|||
|
|
@ -20,10 +20,7 @@ from app.db import (
|
|||
DocumentType,
|
||||
)
|
||||
from app.indexing_pipeline.cache import build_chunk_embeddings
|
||||
from app.indexing_pipeline.cache.cached_indexing import (
|
||||
chunk_markdown_with_lines,
|
||||
embed_batch,
|
||||
)
|
||||
from app.indexing_pipeline.cache.cached_indexing import chunk_markdown, embed_batch
|
||||
from app.indexing_pipeline.chunk_reconciler import ExistingChunk, reconcile
|
||||
from app.indexing_pipeline.connector_document import ConnectorDocument
|
||||
from app.indexing_pipeline.document_hashing import (
|
||||
|
|
@ -62,7 +59,6 @@ from app.indexing_pipeline.pipeline_logger import (
|
|||
log_unexpected_error,
|
||||
)
|
||||
from app.observability import metrics as ot_metrics, otel as ot
|
||||
from app.services.document_revision_recorder import record_prepared_documents
|
||||
from app.utils.perf import get_perf_logger
|
||||
|
||||
|
||||
|
|
@ -338,9 +334,6 @@ class IndexingPipelineService:
|
|||
|
||||
try:
|
||||
await self.session.commit()
|
||||
# Content is durable from here; record it as one revision per batch.
|
||||
# Chunking/embedding failures below never block the record.
|
||||
await record_prepared_documents(self.session, documents)
|
||||
perf.info(
|
||||
"[indexing] prepare_for_indexing in %.3fs input=%d output=%d",
|
||||
time.perf_counter() - t0,
|
||||
|
|
@ -494,22 +487,12 @@ class IndexingPipelineService:
|
|||
|
||||
async def _load_existing_chunks(self, document_id: int) -> list[ExistingChunk]:
|
||||
result = await self.session.execute(
|
||||
select(
|
||||
Chunk.id,
|
||||
Chunk.content,
|
||||
Chunk.position,
|
||||
Chunk.start_line,
|
||||
Chunk.end_line,
|
||||
).where(Chunk.document_id == document_id)
|
||||
select(Chunk.id, Chunk.content, Chunk.position).where(
|
||||
Chunk.document_id == document_id
|
||||
)
|
||||
)
|
||||
return [
|
||||
ExistingChunk(
|
||||
id=row.id,
|
||||
content=row.content,
|
||||
position=row.position,
|
||||
start_line=row.start_line,
|
||||
end_line=row.end_line,
|
||||
)
|
||||
ExistingChunk(id=row.id, content=row.content, position=row.position)
|
||||
for row in result
|
||||
]
|
||||
|
||||
|
|
@ -520,21 +503,15 @@ class IndexingPipelineService:
|
|||
delete(Chunk).where(Chunk.document_id == document.id)
|
||||
)
|
||||
|
||||
summary_embedding, embedded_chunks = await build_chunk_embeddings(
|
||||
summary_embedding, chunk_pairs = await build_chunk_embeddings(
|
||||
content,
|
||||
use_code_chunker=connector_doc.should_use_code_chunker,
|
||||
)
|
||||
|
||||
document.embedding = summary_embedding
|
||||
return [
|
||||
Chunk(
|
||||
content=chunk.text,
|
||||
embedding=chunk.embedding,
|
||||
position=i,
|
||||
start_line=chunk.start_line,
|
||||
end_line=chunk.end_line,
|
||||
)
|
||||
for i, chunk in enumerate(embedded_chunks)
|
||||
Chunk(content=text, embedding=emb, position=i)
|
||||
for i, (text, emb) in enumerate(chunk_pairs)
|
||||
]
|
||||
|
||||
async def _reindex_incrementally(
|
||||
|
|
@ -549,27 +526,19 @@ class IndexingPipelineService:
|
|||
Unchanged rows keep their embedding and their HNSW/GIN index entries;
|
||||
moved rows get a position-only UPDATE, which touches neither index.
|
||||
"""
|
||||
new_chunks = await chunk_markdown_with_lines(
|
||||
new_texts = await chunk_markdown(
|
||||
content, use_code_chunker=connector_doc.should_use_code_chunker
|
||||
)
|
||||
plan = reconcile(existing, new_chunks)
|
||||
plan = reconcile(existing, new_texts)
|
||||
|
||||
# One batch: the document-level summary vector plus the missing chunks.
|
||||
embeddings = await embed_batch([content, *[c.text for c in plan.to_embed]])
|
||||
embeddings = await embed_batch([content, *[t for _, t in plan.to_embed]])
|
||||
summary_embedding, *new_embeddings = embeddings
|
||||
|
||||
if plan.reused:
|
||||
await self.session.execute(
|
||||
update(Chunk),
|
||||
[
|
||||
{
|
||||
"id": reused.id,
|
||||
"position": reused.position,
|
||||
"start_line": reused.start_line,
|
||||
"end_line": reused.end_line,
|
||||
}
|
||||
for reused in plan.reused
|
||||
],
|
||||
[{"id": cid, "position": pos} for cid, pos in plan.reused],
|
||||
)
|
||||
if plan.to_delete:
|
||||
await self.session.execute(
|
||||
|
|
@ -577,14 +546,12 @@ class IndexingPipelineService:
|
|||
)
|
||||
self.session.add_all(
|
||||
Chunk(
|
||||
content=pending.text,
|
||||
content=text,
|
||||
embedding=emb,
|
||||
position=pending.position,
|
||||
start_line=pending.start_line,
|
||||
end_line=pending.end_line,
|
||||
position=pos,
|
||||
document_id=document.id,
|
||||
)
|
||||
for pending, emb in zip(plan.to_embed, new_embeddings, strict=True)
|
||||
for (pos, text), emb in zip(plan.to_embed, new_embeddings, strict=True)
|
||||
)
|
||||
document.embedding = summary_embedding
|
||||
|
||||
|
|
@ -593,7 +560,7 @@ class IndexingPipelineService:
|
|||
embedded=len(plan.to_embed),
|
||||
deleted=len(plan.to_delete),
|
||||
)
|
||||
return len(new_chunks)
|
||||
return len(new_texts)
|
||||
|
||||
async def index_batch_parallel(
|
||||
self,
|
||||
|
|
|
|||
|
|
@ -1,21 +0,0 @@
|
|||
"""Git-native versioned storage for workspace knowledge."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.knowledge_store.engines.base import (
|
||||
Change,
|
||||
Revision,
|
||||
TrackedPath,
|
||||
WorkingCopy,
|
||||
)
|
||||
from app.knowledge_store.store import KnowledgeStore
|
||||
from app.knowledge_store.transaction import Transaction
|
||||
|
||||
__all__ = [
|
||||
"Change",
|
||||
"KnowledgeStore",
|
||||
"Revision",
|
||||
"TrackedPath",
|
||||
"Transaction",
|
||||
"WorkingCopy",
|
||||
]
|
||||
|
|
@ -1,134 +0,0 @@
|
|||
"""Storage-engine contract behind the facade; the seam for swapping engines."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
ChangeKind = Literal["added", "modified", "removed", "renamed"]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Revision:
|
||||
"""One recorded point in a workspace's history (a whole-tree snapshot)."""
|
||||
|
||||
id: str
|
||||
#: Whose content change this is (the acting user).
|
||||
author: str
|
||||
#: Who recorded it (the agent for agent turns; equals author otherwise).
|
||||
committer: str
|
||||
message: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Change:
|
||||
"""One path's change within a revision."""
|
||||
|
||||
path: str
|
||||
kind: ChangeKind
|
||||
#: Content address after the change (``None`` when removed).
|
||||
content_id: str | None
|
||||
#: Where a renamed path came from (``None`` for every other kind).
|
||||
previous_path: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TrackedPath:
|
||||
"""One path stored at a revision, with its content address."""
|
||||
|
||||
path: str
|
||||
content_id: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WorkingCopy:
|
||||
"""A private on-disk copy of the store's content, open for one unit of work."""
|
||||
|
||||
id: str
|
||||
path: Path
|
||||
#: Revision the copy was opened at (``None`` when the store was empty).
|
||||
base_revision: str | None
|
||||
|
||||
|
||||
class VersionedContentEngine(ABC):
|
||||
"""Append-only, content-addressed history for a single workspace.
|
||||
|
||||
First use bootstraps the store; callers never create it explicitly.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def record(
|
||||
self,
|
||||
*,
|
||||
writes: Mapping[str, bytes],
|
||||
removes: Iterable[str],
|
||||
message: str,
|
||||
author: str,
|
||||
committer: str | None = None,
|
||||
) -> str | None:
|
||||
"""Append ``writes`` and ``removes`` to history as one revision.
|
||||
|
||||
``committer`` defaults to ``author``. Returns the revision id, or
|
||||
``None`` when nothing changed.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def read(self, path: str) -> bytes | None:
|
||||
"""Current bytes of ``path``, or ``None`` if it does not exist."""
|
||||
|
||||
@abstractmethod
|
||||
def read_as_of(self, revision: str, path: str) -> bytes:
|
||||
"""Bytes of ``path`` as of ``revision``. Raises if absent there."""
|
||||
|
||||
@abstractmethod
|
||||
def list_revisions(
|
||||
self, *, path: str | None = None, limit: int | None = None
|
||||
) -> list[Revision]:
|
||||
"""Revisions newest-first, optionally scoped to a single path."""
|
||||
|
||||
@abstractmethod
|
||||
def list_changes(self, revision: str, *, since: str | None = None) -> list[Change]:
|
||||
"""What ``revision`` changed, against its parent or against ``since``.
|
||||
|
||||
``since`` compares two snapshots directly, so a path touched repeatedly
|
||||
in between appears once, with its net effect. A path that moved is one
|
||||
``renamed`` change carrying both paths, not a removal plus an addition,
|
||||
so callers can keep whatever they hold against the old path.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def list_paths(self, revision: str) -> list[TrackedPath]:
|
||||
"""Every path stored at ``revision``, with its content address."""
|
||||
|
||||
@abstractmethod
|
||||
def get_current_revision(self) -> str | None:
|
||||
"""Id of the current whole-store snapshot, or ``None`` when empty."""
|
||||
|
||||
@abstractmethod
|
||||
def open_working_copy(self, copy_id: str) -> WorkingCopy:
|
||||
"""Copy of the current content for ``copy_id``; reopens an existing one."""
|
||||
|
||||
@abstractmethod
|
||||
def diff_working_copy(self, copy_id: str) -> tuple[dict[str, bytes], list[str]]:
|
||||
"""Net changes in ``copy_id``'s copy since its base, as ``(writes, removes)``.
|
||||
|
||||
Raises ``FileNotFoundError`` when the copy was never opened.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def discard_working_copy(self, copy_id: str) -> None:
|
||||
"""Delete ``copy_id``'s working copy; a no-op if absent."""
|
||||
|
||||
@abstractmethod
|
||||
def prune_working_copies(self, *, older_than_seconds: float) -> list[str]:
|
||||
"""Delete abandoned working copies; returns the pruned ids."""
|
||||
|
||||
@staticmethod
|
||||
@abstractmethod
|
||||
def compute_content_id(data: bytes) -> str:
|
||||
"""Stable content address for ``data``, independent of any path."""
|
||||
|
|
@ -1,325 +0,0 @@
|
|||
"""Git implementation of the versioned content engine (dulwich).
|
||||
|
||||
Methods are synchronous; the async facade runs them in a worker thread.
|
||||
Public methods are documented on the contract (``engines/base.py``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Iterable, Mapping
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from dulwich import porcelain
|
||||
from dulwich.diff_tree import (
|
||||
CHANGE_ADD,
|
||||
CHANGE_DELETE,
|
||||
CHANGE_MODIFY,
|
||||
CHANGE_RENAME,
|
||||
RenameDetector,
|
||||
tree_changes,
|
||||
)
|
||||
from dulwich.object_store import iter_tree_contents, tree_lookup_path
|
||||
from dulwich.objects import Blob
|
||||
from dulwich.repo import Repo
|
||||
from dulwich.worktree import add_worktree, prune_worktrees, remove_worktree
|
||||
|
||||
from app.knowledge_store.engines.base import (
|
||||
Change,
|
||||
Revision,
|
||||
TrackedPath,
|
||||
VersionedContentEngine,
|
||||
WorkingCopy,
|
||||
)
|
||||
|
||||
_CHANGE_KINDS = {
|
||||
CHANGE_ADD: "added",
|
||||
CHANGE_MODIFY: "modified",
|
||||
CHANGE_DELETE: "removed",
|
||||
CHANGE_RENAME: "renamed",
|
||||
}
|
||||
|
||||
# Serializes working-copy creation against parallel tool calls in one process.
|
||||
# ponytail: one process-wide lock; open is a stat once the copy exists.
|
||||
_open_working_copy_lock = threading.Lock()
|
||||
|
||||
|
||||
class GitContentEngine(VersionedContentEngine):
|
||||
"""One workspace's history as a Git repository at ``path``."""
|
||||
|
||||
def __init__(self, path: Path, working_copies_path: Path) -> None:
|
||||
self._path = path
|
||||
self._working_copies_path = working_copies_path
|
||||
|
||||
def _ensure_exists(self) -> None:
|
||||
"""Bootstrap the repository on first use; a no-op once it exists."""
|
||||
self._path.mkdir(parents=True, exist_ok=True)
|
||||
if not (self._path / ".git").exists():
|
||||
porcelain.init(str(self._path))
|
||||
|
||||
def _exists(self) -> bool:
|
||||
return (self._path / ".git").exists()
|
||||
|
||||
def record(
|
||||
self,
|
||||
*,
|
||||
writes: Mapping[str, bytes],
|
||||
removes: Iterable[str],
|
||||
message: str,
|
||||
author: str,
|
||||
committer: str | None = None,
|
||||
) -> str | None:
|
||||
self._ensure_exists()
|
||||
repo = Repo(str(self._path))
|
||||
try:
|
||||
staged: list[str] = []
|
||||
for rel_path, data in writes.items():
|
||||
abs_path = self._path / rel_path
|
||||
abs_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
abs_path.write_bytes(data)
|
||||
staged.append(str(abs_path))
|
||||
if staged:
|
||||
# One batched add: porcelain.add rewrites the whole index per
|
||||
# call, so per-file adds turn an n-file revision into O(n^2).
|
||||
porcelain.add(repo, paths=staged)
|
||||
|
||||
self._stage_removals(repo, removes)
|
||||
|
||||
if not self._has_pending_changes(repo):
|
||||
return None
|
||||
|
||||
revision = porcelain.commit(
|
||||
repo,
|
||||
message=message.encode(),
|
||||
author=author.encode(),
|
||||
committer=(committer or author).encode(),
|
||||
)
|
||||
return revision.decode()
|
||||
finally:
|
||||
repo.close()
|
||||
|
||||
def read(self, path: str) -> bytes | None:
|
||||
abs_path = self._path / path
|
||||
return abs_path.read_bytes() if abs_path.is_file() else None
|
||||
|
||||
def read_as_of(self, revision: str, path: str) -> bytes:
|
||||
repo = Repo(str(self._path))
|
||||
try:
|
||||
tree_id = repo[revision.encode()].tree
|
||||
_, blob_sha = tree_lookup_path(repo.get_object, tree_id, path.encode())
|
||||
return repo[blob_sha].data
|
||||
finally:
|
||||
repo.close()
|
||||
|
||||
def list_revisions(
|
||||
self, *, path: str | None = None, limit: int | None = None
|
||||
) -> list[Revision]:
|
||||
if not self._exists():
|
||||
return []
|
||||
repo = Repo(str(self._path))
|
||||
try:
|
||||
if repo.head() is None: # pragma: no cover - guarded below
|
||||
return []
|
||||
except KeyError:
|
||||
return []
|
||||
try:
|
||||
walker = repo.get_walker(
|
||||
paths=[path.encode()] if path else None,
|
||||
max_entries=limit,
|
||||
)
|
||||
return [self._to_revision(entry.commit) for entry in walker]
|
||||
finally:
|
||||
repo.close()
|
||||
|
||||
def list_changes(self, revision: str, *, since: str | None = None) -> list[Change]:
|
||||
repo = Repo(str(self._path))
|
||||
try:
|
||||
commit = repo[revision.encode()]
|
||||
base_tree = self._base_tree(repo, commit, since)
|
||||
# Renames come from git's own detection, not a guess of ours: identical
|
||||
# content is matched by hash, so a plain move is always found. Only
|
||||
# similarity matching (a move that also edits) is bounded — dulwich stops
|
||||
# at 200 candidates, past which such a move reads as a removal and an add.
|
||||
detector = RenameDetector(repo.object_store)
|
||||
changes = []
|
||||
for change in tree_changes(
|
||||
repo.object_store, base_tree, commit.tree, rename_detector=detector
|
||||
):
|
||||
kind = _CHANGE_KINDS.get(change.type)
|
||||
if kind is None:
|
||||
continue
|
||||
entry = change.old if kind == "removed" else change.new
|
||||
changes.append(
|
||||
Change(
|
||||
path=entry.path.decode(),
|
||||
kind=kind,
|
||||
content_id=None if kind == "removed" else entry.sha.decode(),
|
||||
previous_path=(
|
||||
change.old.path.decode() if kind == "renamed" else None
|
||||
),
|
||||
)
|
||||
)
|
||||
return changes
|
||||
finally:
|
||||
repo.close()
|
||||
|
||||
def list_paths(self, revision: str) -> list[TrackedPath]:
|
||||
repo = Repo(str(self._path))
|
||||
try:
|
||||
tree_id = repo[revision.encode()].tree
|
||||
return [
|
||||
TrackedPath(path=entry.path.decode(), content_id=entry.sha.decode())
|
||||
for entry in iter_tree_contents(repo.object_store, tree_id)
|
||||
]
|
||||
finally:
|
||||
repo.close()
|
||||
|
||||
def get_current_revision(self) -> str | None:
|
||||
if not self._exists():
|
||||
return None
|
||||
repo = Repo(str(self._path))
|
||||
try:
|
||||
return repo.head().decode()
|
||||
except KeyError:
|
||||
return None
|
||||
finally:
|
||||
repo.close()
|
||||
|
||||
def open_working_copy(self, copy_id: str) -> WorkingCopy:
|
||||
with _open_working_copy_lock:
|
||||
self._ensure_exists()
|
||||
copy_path = self._working_copies_path / copy_id
|
||||
if copy_path.exists():
|
||||
return WorkingCopy(
|
||||
id=copy_id,
|
||||
path=copy_path,
|
||||
base_revision=self._working_copy_base(copy_path),
|
||||
)
|
||||
base = self.get_current_revision()
|
||||
copy_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if base is None:
|
||||
# An empty store has no revision to check out; start from a bare directory.
|
||||
copy_path.mkdir()
|
||||
else:
|
||||
repo = Repo(str(self._path))
|
||||
try:
|
||||
add_worktree(repo, str(copy_path), detach=True).close()
|
||||
finally:
|
||||
repo.close()
|
||||
return WorkingCopy(id=copy_id, path=copy_path, base_revision=base)
|
||||
|
||||
def diff_working_copy(self, copy_id: str) -> tuple[dict[str, bytes], list[str]]:
|
||||
copy_path = self._working_copies_path / copy_id
|
||||
if not copy_path.is_dir():
|
||||
raise FileNotFoundError(f"No working copy '{copy_id}'")
|
||||
if self._working_copy_base(copy_path) is None:
|
||||
return self._all_files_as_writes(copy_path), []
|
||||
|
||||
status = porcelain.status(str(copy_path), untracked_files="all")
|
||||
writes: dict[str, bytes] = {}
|
||||
removes: list[str] = []
|
||||
for raw in status.untracked:
|
||||
rel = raw.decode() if isinstance(raw, bytes) else raw
|
||||
writes[rel] = (copy_path / rel).read_bytes()
|
||||
for raw in status.unstaged:
|
||||
rel = raw.decode() if isinstance(raw, bytes) else raw
|
||||
file = copy_path / rel
|
||||
if file.is_file():
|
||||
writes[rel] = file.read_bytes()
|
||||
else:
|
||||
removes.append(rel)
|
||||
return writes, removes
|
||||
|
||||
def discard_working_copy(self, copy_id: str) -> None:
|
||||
copy_path = self._working_copies_path / copy_id
|
||||
if not copy_path.exists():
|
||||
return
|
||||
if (copy_path / ".git").exists():
|
||||
repo = Repo(str(self._path))
|
||||
try:
|
||||
remove_worktree(repo, str(copy_path), force=True)
|
||||
finally:
|
||||
repo.close()
|
||||
else:
|
||||
shutil.rmtree(copy_path)
|
||||
|
||||
def prune_working_copies(self, *, older_than_seconds: float) -> list[str]:
|
||||
# ponytail: age = the copy directory's own mtime (not nested files), so the
|
||||
# threshold must exceed the longest plausible unit of work by a wide margin.
|
||||
if not self._working_copies_path.exists():
|
||||
return []
|
||||
cutoff = time.time() - older_than_seconds
|
||||
pruned = [
|
||||
entry.name
|
||||
for entry in self._working_copies_path.iterdir()
|
||||
if entry.is_dir() and entry.stat().st_mtime < cutoff
|
||||
]
|
||||
for copy_id in pruned:
|
||||
self.discard_working_copy(copy_id)
|
||||
if (self._path / ".git").exists():
|
||||
repo = Repo(str(self._path))
|
||||
try:
|
||||
# Drop bookkeeping left by copies whose directory vanished (e.g. a crash).
|
||||
prune_worktrees(repo, expire=0)
|
||||
finally:
|
||||
repo.close()
|
||||
return pruned
|
||||
|
||||
@staticmethod
|
||||
def compute_content_id(data: bytes) -> str:
|
||||
return Blob.from_string(data).id.decode()
|
||||
|
||||
@staticmethod
|
||||
def _base_tree(repo: Repo, commit, since: str | None):
|
||||
"""What to diff against: ``since``'s tree, else the commit's first parent."""
|
||||
if since is not None:
|
||||
return repo[since.encode()].tree
|
||||
return repo[commit.parents[0]].tree if commit.parents else None
|
||||
|
||||
@staticmethod
|
||||
def _working_copy_base(copy_path: Path) -> str | None:
|
||||
"""Revision an existing copy was opened at (``None`` for a bare directory)."""
|
||||
if not (copy_path / ".git").exists():
|
||||
return None
|
||||
repo = Repo(str(copy_path))
|
||||
try:
|
||||
return repo.head().decode()
|
||||
finally:
|
||||
repo.close()
|
||||
|
||||
@staticmethod
|
||||
def _all_files_as_writes(copy_path: Path) -> dict[str, bytes]:
|
||||
return {
|
||||
str(file.relative_to(copy_path)): file.read_bytes()
|
||||
for file in sorted(copy_path.rglob("*"))
|
||||
if file.is_file() and ".git" not in file.relative_to(copy_path).parts
|
||||
}
|
||||
|
||||
def _stage_removals(self, repo: Repo, rel_paths: Iterable[str]) -> None:
|
||||
"""Stage deletions in one batch, tolerating never-tracked paths."""
|
||||
index = repo.open_index()
|
||||
tracked = [p for p in rel_paths if p.encode() in index]
|
||||
if tracked:
|
||||
porcelain.remove(repo, paths=[str(self._path / p) for p in tracked])
|
||||
|
||||
def _has_pending_changes(self, repo: Repo) -> bool:
|
||||
"""Whether the staged index differs from the current head tree."""
|
||||
index_tree = repo.open_index().commit(repo.object_store)
|
||||
try:
|
||||
head_tree = repo[repo.head()].tree
|
||||
except KeyError:
|
||||
return True
|
||||
return index_tree != head_tree
|
||||
|
||||
@staticmethod
|
||||
def _to_revision(commit) -> Revision:
|
||||
return Revision(
|
||||
id=commit.id.decode(),
|
||||
author=commit.author.decode(),
|
||||
committer=commit.committer.decode(),
|
||||
message=commit.message.decode().strip(),
|
||||
created_at=datetime.fromtimestamp(commit.commit_time, tz=UTC),
|
||||
)
|
||||
|
|
@ -1,19 +0,0 @@
|
|||
"""Authorship conventions for recorded revisions.
|
||||
|
||||
Author = whose content change it is; committer = who recorded it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
#: Committer of every agent-recorded revision; author of autonomous writes.
|
||||
AGENT_IDENTITY = "SurfSense Agent <agent@surfsense>"
|
||||
|
||||
#: Author of the one-time seed revision that migrates a workspace into the store.
|
||||
MIGRATION_IDENTITY = "SurfSense Migration <migration@surfsense>"
|
||||
|
||||
|
||||
def user_identity(user_id: str | None) -> str:
|
||||
"""Revision author for a user action; autonomous actions author as the agent."""
|
||||
if user_id is None:
|
||||
return AGENT_IDENTITY
|
||||
return f"SurfSense User <{user_id}@users.surfsense>"
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
"""The derived Postgres index of the store, and the queue that drives it.
|
||||
|
||||
Deliberately re-exports nothing. :mod:`.converge` reaches into ``app.db``, the
|
||||
indexing pipeline and the agents middleware, while :mod:`.queue` is a writer's
|
||||
last step and must stay cheap to import; a convenience re-export here would put
|
||||
the former on the latter's import path.
|
||||
|
||||
Everything in this subpackage is a **driven consumer** of the store (ADR 0002):
|
||||
it subscribes to revisions one way, and the core never imports it back.
|
||||
"""
|
||||
|
|
@ -1,490 +0,0 @@
|
|||
"""Postgres as a derived index of the store.
|
||||
|
||||
Git holds the content; ``documents`` + ``chunks`` are a rebuildable projection of
|
||||
it. The two entry points differ only in scope: :func:`index_changes` folds in
|
||||
what moved since the last run, :func:`index_tree` reconciles against the whole
|
||||
tree and is therefore the only one that can notice a deletion it never saw.
|
||||
Both run the same convergence body, so the two paths cannot drift apart.
|
||||
|
||||
Neither wipes. Document rows are upserted by path and keep their ids, because
|
||||
``documents``/``folders`` replicate to the browser and an id that changed under
|
||||
a reader would make every note vanish and reappear. Chunk rows are the
|
||||
disposable layer, replaced per document by the existing indexing pipeline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.agents.chat.multi_agent_chat.main_agent.middleware.kb_persistence.middleware import (
|
||||
ensure_folder_hierarchy,
|
||||
)
|
||||
from app.agents.chat.runtime.path_resolver import (
|
||||
PATH_MARKER,
|
||||
parse_documents_path,
|
||||
to_virtual_path,
|
||||
virtual_path_to_doc,
|
||||
)
|
||||
from app.db import Document, DocumentStatus, DocumentType, Workspace
|
||||
from app.indexing_pipeline.connector_document import ConnectorDocument
|
||||
from app.indexing_pipeline.indexing_pipeline_service import IndexingPipelineService
|
||||
from app.knowledge_store.engines.base import Change
|
||||
from app.knowledge_store.store import KnowledgeStore
|
||||
from app.knowledge_store.write_lock import workspace_index_lock
|
||||
from app.utils.document_converters import (
|
||||
generate_content_hash,
|
||||
generate_unique_identifier_hash,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# PATH_MARKER marks a row as living at a store path, i.e. owned by this indexer.
|
||||
# Rows without it (Slack, Notion, the folder indexers) are never pruned.
|
||||
|
||||
_USER_AUTHOR = re.compile(r"<([^@>]+)@users\.surfsense>")
|
||||
|
||||
|
||||
@dataclass
|
||||
class IndexOutcome:
|
||||
"""What one convergence run did, and whether it may stamp the revision."""
|
||||
|
||||
revision: str | None
|
||||
indexed: int = 0
|
||||
skipped: int = 0
|
||||
failed: int = 0
|
||||
deleted: int = 0
|
||||
stamped: bool = False
|
||||
|
||||
def __str__(self) -> str:
|
||||
return (
|
||||
f"revision={self.revision} indexed={self.indexed} "
|
||||
f"skipped={self.skipped} failed={self.failed} "
|
||||
f"deleted={self.deleted} stamped={self.stamped}"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Plan:
|
||||
"""Which store paths to converge, and whether to prune to the whole tree."""
|
||||
|
||||
upserts: list[str]
|
||||
removals: list[str]
|
||||
#: Paths that moved, as ``(from, to)``; the row follows instead of being remade.
|
||||
renames: list[tuple[str, str]] = field(default_factory=list)
|
||||
#: Every path in the tree, when the run is a full rebuild; ``None`` otherwise.
|
||||
tree: set[str] | None = field(default=None)
|
||||
|
||||
|
||||
async def index_changes(session: AsyncSession, workspace_id: int) -> IndexOutcome:
|
||||
"""Fold the paths that moved since the last run into the index.
|
||||
|
||||
Always converges to whatever HEAD is now, never to the revision that
|
||||
triggered the call: two saves in a row enqueue two tasks, the index lock
|
||||
serializes them without ordering them, and stamping the older id last would
|
||||
leave a stale index. Reading HEAD under the lock makes task order
|
||||
irrelevant, which is why no revision is passed in.
|
||||
"""
|
||||
return await _run(session, workspace_id, full=False)
|
||||
|
||||
|
||||
async def index_tree(session: AsyncSession, workspace_id: int) -> IndexOutcome:
|
||||
"""Reconcile the index against every path in the tree (the Fossil rebuild).
|
||||
|
||||
Distrusts the stamp, so this is the only path that removes a row whose file
|
||||
left the tree while the index was not watching — and the only repair for an
|
||||
index that fell behind in a way the change log can no longer describe.
|
||||
"""
|
||||
return await _run(session, workspace_id, full=True)
|
||||
|
||||
|
||||
async def _run(session: AsyncSession, workspace_id: int, *, full: bool) -> IndexOutcome:
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
async with workspace_index_lock(workspace_id):
|
||||
head = await store.get_current_revision()
|
||||
if head is None:
|
||||
return IndexOutcome(revision=None)
|
||||
|
||||
workspace = await session.get(Workspace, workspace_id)
|
||||
if workspace is None:
|
||||
logger.warning("Workspace %s no longer exists; not indexing", workspace_id)
|
||||
return IndexOutcome(revision=head)
|
||||
if not full and workspace.last_indexed_revision == head:
|
||||
return IndexOutcome(revision=head, stamped=True)
|
||||
|
||||
since = None if full else workspace.last_indexed_revision
|
||||
plan = await _plan(store, head, since)
|
||||
outcome = await _converge(session, store, workspace, head, plan)
|
||||
|
||||
logger.info("Knowledge store index for workspace %s: %s", workspace_id, outcome)
|
||||
return outcome
|
||||
|
||||
|
||||
async def _plan(store: KnowledgeStore, head: str, since: str | None) -> _Plan:
|
||||
"""Paths to converge: the changes since ``since``, else the whole tree."""
|
||||
if since is not None:
|
||||
changes = await _changes_since(store, head, since)
|
||||
if changes is not None:
|
||||
return _Plan(
|
||||
upserts=[c.path for c in changes if c.kind != "removed"],
|
||||
removals=[c.path for c in changes if c.kind == "removed"],
|
||||
renames=[
|
||||
(c.previous_path, c.path)
|
||||
for c in changes
|
||||
if c.kind == "renamed" and c.previous_path
|
||||
],
|
||||
)
|
||||
tracked = [entry.path for entry in await store.list_paths(head)]
|
||||
return _Plan(upserts=tracked, removals=[], tree=set(tracked))
|
||||
|
||||
|
||||
async def _changes_since(
|
||||
store: KnowledgeStore, head: str, since: str
|
||||
) -> list[Change] | None:
|
||||
"""Net change set from ``since`` (exclusive) to ``head``.
|
||||
|
||||
One diff of the two snapshots rather than a fold of each revision between
|
||||
them, which matters because a queued task can be several commits behind by
|
||||
the time it runs: a path written twice in that window appears once, a path
|
||||
written then deleted appears not at all, and a move is a single ``renamed``
|
||||
change that keeps both of its paths.
|
||||
|
||||
``None`` when ``since`` is not in the history any more, which asks the caller
|
||||
for a full rebuild rather than a guess.
|
||||
|
||||
``ponytail:`` walks the whole revision list to locate ``since``; upgrade path
|
||||
is a bounded walk once histories get long enough to notice.
|
||||
"""
|
||||
ids = [revision.id for revision in await store.list_revisions()]
|
||||
if since not in ids:
|
||||
return None
|
||||
return await store.list_changes(head, since=since)
|
||||
|
||||
|
||||
async def _converge(
|
||||
session: AsyncSession,
|
||||
store: KnowledgeStore,
|
||||
workspace: Workspace,
|
||||
head: str,
|
||||
plan: _Plan,
|
||||
) -> IndexOutcome:
|
||||
outcome = IndexOutcome(revision=head)
|
||||
owned = await _load_owned(session, workspace.id)
|
||||
author_id = await _revision_author_id(store, head, workspace)
|
||||
|
||||
for from_path, to_path in plan.renames:
|
||||
_follow_rename(
|
||||
owned,
|
||||
workspace.id,
|
||||
to_virtual_path(from_path),
|
||||
to_virtual_path(to_path),
|
||||
)
|
||||
|
||||
for store_path in plan.upserts:
|
||||
virtual_path = to_virtual_path(store_path)
|
||||
content = await _read_indexable(store, head, store_path)
|
||||
if content is None:
|
||||
outcome.skipped += 1
|
||||
continue
|
||||
ready = await _index_one(
|
||||
session,
|
||||
workspace_id=workspace.id,
|
||||
virtual_path=virtual_path,
|
||||
content=content,
|
||||
author_id=author_id,
|
||||
owned=owned,
|
||||
)
|
||||
if ready:
|
||||
outcome.indexed += 1
|
||||
else:
|
||||
outcome.failed += 1
|
||||
|
||||
for store_path in plan.removals:
|
||||
outcome.deleted += await _delete(
|
||||
session, workspace.id, to_virtual_path(store_path), owned
|
||||
)
|
||||
|
||||
if plan.tree is not None:
|
||||
live = {to_virtual_path(path) for path in plan.tree}
|
||||
outcome.deleted += await _prune(session, owned, live)
|
||||
|
||||
# A failed document must not advance the marker, or the drift sweep can never
|
||||
# re-drive it. An intentional skip (unreadable blob) must not block it, or one
|
||||
# bad file wedges the workspace into rebuilding itself forever.
|
||||
if outcome.failed == 0:
|
||||
workspace.last_indexed_revision = head
|
||||
outcome.stamped = True
|
||||
await session.commit()
|
||||
return outcome
|
||||
|
||||
|
||||
async def _read_indexable(
|
||||
store: KnowledgeStore, revision: str, store_path: str
|
||||
) -> str | None:
|
||||
"""Decoded, non-blank text of a blob, or ``None`` when it can't be indexed.
|
||||
|
||||
One unusable blob must not strand every other document in the revision, and
|
||||
both cases here are legal git: ``touch``ed files and binaries.
|
||||
"""
|
||||
try:
|
||||
raw = await store.read_as_of(revision, store_path)
|
||||
except Exception:
|
||||
logger.warning("Skipping unreadable path %s", store_path, exc_info=True)
|
||||
return None
|
||||
try:
|
||||
content = raw.decode()
|
||||
except UnicodeDecodeError:
|
||||
logger.info("Skipping undecodable blob at %s", store_path)
|
||||
return None
|
||||
if not content.strip():
|
||||
logger.info("Skipping blank document at %s", store_path)
|
||||
return None
|
||||
return content
|
||||
|
||||
|
||||
async def _index_one(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
workspace_id: int,
|
||||
virtual_path: str,
|
||||
content: str,
|
||||
author_id: str,
|
||||
owned: dict[str, Document],
|
||||
) -> bool:
|
||||
"""Upsert the row for one path, then hand it to the indexing pipeline."""
|
||||
folder_parts, title = parse_documents_path(virtual_path)
|
||||
if not title:
|
||||
logger.info("Skipping path with no document name: %s", virtual_path)
|
||||
return True
|
||||
|
||||
document = await _resolve(session, workspace_id, virtual_path, owned)
|
||||
folder_id = await ensure_folder_hierarchy(
|
||||
session,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=author_id,
|
||||
folder_parts=folder_parts,
|
||||
)
|
||||
metadata = {**(document.document_metadata or {} if document else {})}
|
||||
metadata[PATH_MARKER] = virtual_path
|
||||
|
||||
if document is None:
|
||||
document = Document(
|
||||
title=title,
|
||||
document_type=DocumentType.NOTE,
|
||||
document_metadata=metadata,
|
||||
content=content,
|
||||
content_hash=generate_content_hash(content, workspace_id),
|
||||
unique_identifier_hash=generate_unique_identifier_hash(
|
||||
DocumentType.NOTE, virtual_path, workspace_id
|
||||
),
|
||||
source_markdown=content,
|
||||
workspace_id=workspace_id,
|
||||
folder_id=folder_id,
|
||||
created_by_id=author_id,
|
||||
status=DocumentStatus.pending(),
|
||||
updated_at=datetime.now(UTC),
|
||||
)
|
||||
session.add(document)
|
||||
else:
|
||||
# Update in place. No collision guard here: a hash hit is this path's
|
||||
# normal update case, not an error.
|
||||
document.title = title
|
||||
document.folder_id = folder_id
|
||||
document.source_markdown = content
|
||||
document.content_hash = generate_content_hash(content, workspace_id)
|
||||
document.document_metadata = metadata
|
||||
document.updated_at = datetime.now(UTC)
|
||||
|
||||
# index() needs a persisted row and its id.
|
||||
await session.flush()
|
||||
|
||||
connector_doc = ConnectorDocument(
|
||||
title=title,
|
||||
source_markdown=content,
|
||||
unique_id=virtual_path,
|
||||
document_type=document.document_type,
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=str(document.created_by_id or author_id),
|
||||
connector_id=document.connector_id,
|
||||
metadata=metadata,
|
||||
folder_id=folder_id,
|
||||
)
|
||||
indexed = await IndexingPipelineService(session).index(document, connector_doc)
|
||||
if not DocumentStatus.is_state(indexed.status, DocumentStatus.READY):
|
||||
logger.warning(
|
||||
"Indexing failed for %s: %s",
|
||||
virtual_path,
|
||||
(indexed.status or {}).get("reason"),
|
||||
)
|
||||
# index() rolls back on failure, which un-persists a row this run created.
|
||||
# Leaving it in the owned set would hand prune a transient object.
|
||||
owned.pop(virtual_path, None)
|
||||
return False
|
||||
|
||||
# Recorded only once index() has committed it, so every entry is a real row.
|
||||
owned[virtual_path] = document
|
||||
return True
|
||||
|
||||
|
||||
def _follow_rename(
|
||||
owned: dict[str, Document],
|
||||
workspace_id: int,
|
||||
from_virtual: str,
|
||||
to_virtual: str,
|
||||
) -> None:
|
||||
"""Point the row living at ``from_virtual`` at the path it moved to.
|
||||
|
||||
A move has to leave the row's id alone: ``document_versions`` and an upload's
|
||||
stored original both cascade from it, and citations saved in earlier answers
|
||||
name it. Re-keying is the whole trick — the upsert of the new path then
|
||||
resolves to this row and updates it in place, rather than inserting one row
|
||||
and deleting the other.
|
||||
"""
|
||||
document = owned.pop(from_virtual, None)
|
||||
if document is None:
|
||||
# Nothing marked at the old path: an unindexed file, or a recorder that
|
||||
# already moved the marker. Either way the upsert resolves it by itself.
|
||||
return
|
||||
owned[to_virtual] = document
|
||||
from_hash = generate_unique_identifier_hash(
|
||||
DocumentType.NOTE, from_virtual, workspace_id
|
||||
)
|
||||
if document.unique_identifier_hash == from_hash:
|
||||
# Carry _resolve's fallback key along with the marker, or a later file at
|
||||
# the old path resolves to this row. Only when the key is the path's own:
|
||||
# an upload identifies by filename, and rewriting that would let a
|
||||
# re-upload of the same file insert a second row.
|
||||
document.unique_identifier_hash = generate_unique_identifier_hash(
|
||||
DocumentType.NOTE, to_virtual, workspace_id
|
||||
)
|
||||
|
||||
|
||||
async def _resolve(
|
||||
session: AsyncSession,
|
||||
workspace_id: int,
|
||||
virtual_path: str,
|
||||
owned: dict[str, Document],
|
||||
) -> Document | None:
|
||||
"""Find the row that already represents ``virtual_path``, if any.
|
||||
|
||||
Uploads reach git through the recorder while their row keeps the identity the
|
||||
upload gave it (``FILE:<filename>``), so a NOTE-hash-only lookup would insert
|
||||
a second row for content that already has one — the same file twice in the
|
||||
tree and twice in search. Adopt whatever is already there instead.
|
||||
"""
|
||||
marked = owned.get(virtual_path)
|
||||
if marked is not None:
|
||||
return marked
|
||||
|
||||
unique_hash = generate_unique_identifier_hash(
|
||||
DocumentType.NOTE, virtual_path, workspace_id
|
||||
)
|
||||
result = await session.execute(
|
||||
select(Document).where(
|
||||
Document.workspace_id == workspace_id,
|
||||
Document.unique_identifier_hash == unique_hash,
|
||||
)
|
||||
)
|
||||
document = result.scalar_one_or_none()
|
||||
if document is not None:
|
||||
return document
|
||||
|
||||
return await virtual_path_to_doc(
|
||||
session, workspace_id=workspace_id, virtual_path=virtual_path
|
||||
)
|
||||
|
||||
|
||||
async def _delete(
|
||||
session: AsyncSession,
|
||||
workspace_id: int,
|
||||
virtual_path: str,
|
||||
owned: dict[str, Document],
|
||||
) -> int:
|
||||
"""Drop the document at a removed path; its chunks cascade."""
|
||||
document = await _resolve(session, workspace_id, virtual_path, owned)
|
||||
if document is None:
|
||||
return 0
|
||||
marker = (document.document_metadata or {}).get(PATH_MARKER)
|
||||
if marker and marker != virtual_path:
|
||||
# The row moved, it did not go away: the upsert has already claimed it, so
|
||||
# deleting here would drop what this same run just wrote. Reached when git
|
||||
# cannot see the move — a rewrite in flight leaves nothing to match, so it
|
||||
# arrives as a removal and an addition — while the recorder has moved the
|
||||
# marker and left unique_identifier_hash, _resolve's fallback, behind.
|
||||
return 0
|
||||
owned.pop(virtual_path, None)
|
||||
await session.delete(document)
|
||||
return 1
|
||||
|
||||
|
||||
async def _prune(
|
||||
session: AsyncSession, owned: dict[str, Document], live: set[str]
|
||||
) -> int:
|
||||
"""Delete indexer-owned rows whose path is no longer in the tree.
|
||||
|
||||
Scoped to the marker, never to the workspace: connector rows (Slack, Notion,
|
||||
the folder indexers) have no path in the tree at all, and a workspace-wide
|
||||
prune would delete every one of them on the first rebuild.
|
||||
"""
|
||||
deleted = 0
|
||||
for virtual_path, document in list(owned.items()):
|
||||
if virtual_path in live:
|
||||
continue
|
||||
await session.delete(document)
|
||||
owned.pop(virtual_path, None)
|
||||
deleted += 1
|
||||
return deleted
|
||||
|
||||
|
||||
async def _load_owned(session: AsyncSession, workspace_id: int) -> dict[str, Document]:
|
||||
"""Indexer-owned rows for a workspace, keyed by the path they live at."""
|
||||
result = await session.execute(
|
||||
select(Document).where(
|
||||
Document.workspace_id == workspace_id,
|
||||
Document.document_metadata[PATH_MARKER].as_string().is_not(None),
|
||||
)
|
||||
)
|
||||
owned: dict[str, Document] = {}
|
||||
for document in result.scalars():
|
||||
marker = (document.document_metadata or {}).get(PATH_MARKER)
|
||||
if marker:
|
||||
owned[marker] = document
|
||||
return owned
|
||||
|
||||
|
||||
async def _revision_author_id(
|
||||
store: KnowledgeStore, revision: str, workspace: Workspace
|
||||
) -> str:
|
||||
"""Actor for rows this run creates, derived from git — never passed in.
|
||||
|
||||
A caller-supplied id would be erased by the next :func:`index_tree`, making
|
||||
the two paths disagree. Autonomous agent writes author as the agent, which carries no
|
||||
user id, so those fall back to the workspace owner: ``created_by_id`` is
|
||||
required and rejects blanks, and agent writes are the whole point of indexing.
|
||||
"""
|
||||
owner = str(workspace.user_id)
|
||||
try:
|
||||
revisions = await store.list_revisions(limit=1)
|
||||
except Exception:
|
||||
logger.warning("Could not read revision author for %s", revision, exc_info=True)
|
||||
return owner
|
||||
if not revisions:
|
||||
return owner
|
||||
return _author_user_id(revisions[0].author) or owner
|
||||
|
||||
|
||||
def _author_user_id(author: str) -> str | None:
|
||||
"""User id encoded in a revision author, or ``None`` for the agent."""
|
||||
match = _USER_AUTHOR.search(author or "")
|
||||
if match is None:
|
||||
return None
|
||||
try:
|
||||
return str(uuid.UUID(match.group(1)))
|
||||
except ValueError:
|
||||
return None
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
"""The one way a writer asks for its revision to be indexed.
|
||||
|
||||
Both store writers (the editor/upload recorder and the end-of-turn agent commit)
|
||||
call :func:`enqueue_index` after their content is already committed, so a broker
|
||||
problem is logged and dropped rather than failing a save that succeeded. The
|
||||
hourly drift sweep is the backstop for anything lost that way.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from app.knowledge_store.settings import load_knowledge_store_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def enqueue_index(workspace_id: int | str) -> None:
|
||||
"""Ask a worker to fold the store's current revision into the index.
|
||||
|
||||
Only the global kill switch is checked here (sync context, cheap): every
|
||||
caller already resolved the per-workspace flag before writing, and the
|
||||
worker task re-checks it before converging.
|
||||
"""
|
||||
if not load_knowledge_store_settings().enabled:
|
||||
return
|
||||
try:
|
||||
numeric = int(workspace_id)
|
||||
except (TypeError, ValueError):
|
||||
# Non-numeric ids exist only in tests, which have no worker to serve them.
|
||||
return
|
||||
try:
|
||||
# Imported here: the task module imports the indexer, which imports the
|
||||
# store — a module-level import would close that loop.
|
||||
from app.tasks.celery_tasks.knowledge_store.index_tasks import (
|
||||
index_knowledge_store_revision,
|
||||
)
|
||||
|
||||
index_knowledge_store_revision.delay(numeric)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"Could not enqueue indexing for workspace %s; "
|
||||
"the drift sweep will pick it up",
|
||||
workspace_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
"""Sweep abandoned working copies across every workspace."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.knowledge_store.store import KnowledgeStore
|
||||
from app.knowledge_store.store_path import working_copies_root
|
||||
|
||||
#: Far beyond any turn; a crashed turn's copy is recovered (committed) by the
|
||||
#: thread's next turn well before this — only abandoned threads reach the TTL.
|
||||
DEFAULT_MAX_AGE_SECONDS = 24 * 60 * 60
|
||||
|
||||
|
||||
async def prune_abandoned_working_copies(
|
||||
*, older_than_seconds: float = DEFAULT_MAX_AGE_SECONDS
|
||||
) -> dict[str, list[str]]:
|
||||
"""Prune copies older than the TTL in every workspace; pruned ids by workspace."""
|
||||
root = working_copies_root()
|
||||
if not root.is_dir():
|
||||
return {}
|
||||
pruned: dict[str, list[str]] = {}
|
||||
for workspace_dir in sorted(root.iterdir()):
|
||||
if not workspace_dir.is_dir():
|
||||
continue
|
||||
store = KnowledgeStore.for_workspace(workspace_dir.name)
|
||||
ids = await store.prune_working_copies(older_than_seconds=older_than_seconds)
|
||||
if ids:
|
||||
pruned[workspace_dir.name] = ids
|
||||
return pruned
|
||||
|
|
@ -1,236 +0,0 @@
|
|||
"""Phase 5 seeder: export a workspace's documents into its store as one seed revision.
|
||||
|
||||
Parity is **byte identity** (content addresses compared, no file reads), never a
|
||||
reindex — the seed copies bytes out of Postgres, so the existing chunk index is
|
||||
already correct by construction. Runs before the flip, so unlike the recorder it
|
||||
never guards on ``KNOWLEDGE_STORE_ENABLED``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.agents.chat.runtime.path_resolver import (
|
||||
PATH_MARKER,
|
||||
build_path_index,
|
||||
to_store_path,
|
||||
virtual_path_of,
|
||||
)
|
||||
from app.knowledge_store.engines.base import TrackedPath
|
||||
from app.knowledge_store.identities import MIGRATION_IDENTITY
|
||||
from app.knowledge_store.store import KnowledgeStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MigrationReport:
|
||||
"""Outcome of one seed run; ``ok`` is the flip guard's verdict."""
|
||||
|
||||
workspace_id: int | str
|
||||
dry_run: bool
|
||||
#: Revision recorded by this run; ``None`` on dry runs and no-op re-seeds.
|
||||
seeded_revision: str | None
|
||||
files: int
|
||||
missing: list[str] = field(default_factory=list)
|
||||
extra: list[str] = field(default_factory=list)
|
||||
mismatched: list[str] = field(default_factory=list)
|
||||
#: Failure this run captured instead of raising (e.g. an expired write
|
||||
#: lock); the parity fields describe whatever could still be inspected.
|
||||
error: str | None = None
|
||||
|
||||
@property
|
||||
def ok(self) -> bool:
|
||||
return self.error is None and not (
|
||||
self.missing or self.extra or self.mismatched
|
||||
)
|
||||
|
||||
|
||||
async def seed_workspace(
|
||||
workspace_id: int | str,
|
||||
files: Mapping[str, str],
|
||||
*,
|
||||
dry_run: bool = False,
|
||||
) -> MigrationReport:
|
||||
"""Record ``files`` (store path → markdown) as one seed revision, then verify.
|
||||
|
||||
Idempotent: re-seeding unchanged content records nothing. ``dry_run`` skips
|
||||
the write and only reports parity of ``files`` against the store's head.
|
||||
|
||||
Never raises: any failure is returned as ``MigrationReport.error`` so a
|
||||
fleet-wide run records it and continues with the next workspace.
|
||||
"""
|
||||
try:
|
||||
return await _seed_and_verify(workspace_id, files, dry_run=dry_run)
|
||||
except Exception as exc:
|
||||
return _failure_report(workspace_id, dry_run, len(files), exc)
|
||||
|
||||
|
||||
async def _seed_and_verify(
|
||||
workspace_id: int | str,
|
||||
files: Mapping[str, str],
|
||||
*,
|
||||
dry_run: bool,
|
||||
) -> MigrationReport:
|
||||
"""``seed_workspace``'s body; raises freely, the wrapper reports."""
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
|
||||
seeded_revision: str | None = None
|
||||
error: str | None = None
|
||||
if not dry_run and files:
|
||||
try:
|
||||
# Seed = "make the tree exactly this": orphans from documents
|
||||
# deleted in Postgres since a prior seed are removed, so
|
||||
# re-seeding converges.
|
||||
orphans = [
|
||||
t.path for t in await _tracked_paths(store) if t.path not in files
|
||||
]
|
||||
async with store.transaction(
|
||||
message=f"migration: seed {len(files)} document(s)",
|
||||
author=MIGRATION_IDENTITY,
|
||||
) as tx:
|
||||
for path, markdown in files.items():
|
||||
tx.write(path, markdown.encode())
|
||||
for path in orphans:
|
||||
tx.remove(path)
|
||||
seeded_revision = tx.revision
|
||||
except Exception as exc:
|
||||
# Caught here, not by the wrapper, so parity still runs and the
|
||||
# report shows what state the failed write left behind.
|
||||
error = f"{type(exc).__name__}: {exc}"
|
||||
|
||||
tracked = {t.path: t.content_id for t in await _tracked_paths(store)}
|
||||
desired = {
|
||||
path: store.compute_content_id(markdown.encode())
|
||||
for path, markdown in files.items()
|
||||
}
|
||||
|
||||
return MigrationReport(
|
||||
workspace_id=workspace_id,
|
||||
dry_run=dry_run,
|
||||
seeded_revision=seeded_revision,
|
||||
files=len(desired),
|
||||
missing=sorted(p for p in desired if p not in tracked),
|
||||
extra=sorted(p for p in tracked if p not in desired),
|
||||
mismatched=sorted(
|
||||
p for p, cid in desired.items() if p in tracked and tracked[p] != cid
|
||||
),
|
||||
error=error,
|
||||
)
|
||||
|
||||
|
||||
def _failure_report(
|
||||
workspace_id: int | str, dry_run: bool, files: int, exc: Exception
|
||||
) -> MigrationReport:
|
||||
"""One workspace's failure as an outcome, so a fleet run can move on."""
|
||||
return MigrationReport(
|
||||
workspace_id=workspace_id,
|
||||
dry_run=dry_run,
|
||||
seeded_revision=None,
|
||||
files=files,
|
||||
error=f"{type(exc).__name__}: {exc}",
|
||||
)
|
||||
|
||||
|
||||
async def _tracked_paths(store: KnowledgeStore) -> list[TrackedPath]:
|
||||
"""Paths at the store's head; empty for a store with no history yet."""
|
||||
head = await store.get_current_revision()
|
||||
return await store.list_paths(head) if head else []
|
||||
|
||||
|
||||
async def migrate_workspace(
|
||||
session: AsyncSession,
|
||||
workspace_id: int,
|
||||
*,
|
||||
dry_run: bool = False,
|
||||
) -> MigrationReport:
|
||||
"""Seed one workspace's current documents at the paths they already live at.
|
||||
|
||||
Placement must agree with every other writer, or one document forks into two
|
||||
files. `virtual_path_of` reads the path a row already records and derives from
|
||||
the title only for rows that have none — the agent's `write_file` names its
|
||||
own files, so derivation alone disagrees with the store for anything it
|
||||
authored.
|
||||
|
||||
A successful run records each seeded path back onto its row, so the retitle
|
||||
that follows knows which file to drop from the tree.
|
||||
|
||||
Never raises: a failure while fetching or mapping documents is returned
|
||||
as ``MigrationReport.error``, like every seed failure.
|
||||
"""
|
||||
from app.db import Document
|
||||
|
||||
try:
|
||||
index = await build_path_index(session, workspace_id)
|
||||
rows = await session.execute(
|
||||
select(
|
||||
Document.id,
|
||||
Document.title,
|
||||
Document.folder_id,
|
||||
Document.document_metadata,
|
||||
Document.source_markdown,
|
||||
Document.content,
|
||||
).where(Document.workspace_id == workspace_id)
|
||||
)
|
||||
files: dict[str, str] = {}
|
||||
seeded_paths: dict[int, str] = {}
|
||||
for doc_id, title, folder_id, metadata, source_markdown, content in rows:
|
||||
# Rows predating the nullable source_markdown column hold text in
|
||||
# content only; "Pending..." is the pre-index placeholder, never
|
||||
# content.
|
||||
markdown = source_markdown or content
|
||||
if not markdown or markdown == "Pending...":
|
||||
continue
|
||||
virtual_path = virtual_path_of(
|
||||
metadata=metadata,
|
||||
doc_id=doc_id,
|
||||
title=title,
|
||||
folder_id=folder_id,
|
||||
index=index,
|
||||
)
|
||||
files[to_store_path(virtual_path)] = markdown
|
||||
seeded_paths[doc_id] = virtual_path
|
||||
except Exception as exc:
|
||||
return _failure_report(workspace_id, dry_run, 0, exc)
|
||||
|
||||
report = await seed_workspace(workspace_id, files, dry_run=dry_run)
|
||||
if report.ok and not dry_run:
|
||||
await _record_seeded_paths(session, seeded_paths)
|
||||
return report
|
||||
|
||||
|
||||
async def _record_seeded_paths(
|
||||
session: AsyncSession, seeded_paths: Mapping[int, str]
|
||||
) -> None:
|
||||
"""Mark each seeded row with the path its content was written to.
|
||||
|
||||
Only rows whose marker would change are touched, so a re-seed of an already
|
||||
marked workspace writes nothing. Best-effort: the seed revision is already
|
||||
committed, and an unmarked row still resolves by derivation — it just cannot
|
||||
survive a retitle, which the next seed repairs.
|
||||
"""
|
||||
from app.db import Document
|
||||
|
||||
if not seeded_paths:
|
||||
return
|
||||
try:
|
||||
rows = await session.execute(
|
||||
select(Document).where(Document.id.in_(list(seeded_paths)))
|
||||
)
|
||||
for document in rows.scalars().all():
|
||||
path = seeded_paths[document.id]
|
||||
metadata = dict(document.document_metadata or {})
|
||||
if metadata.get(PATH_MARKER) == path:
|
||||
continue
|
||||
metadata[PATH_MARKER] = path
|
||||
# Reassigned, not mutated: SQLAlchemy tracks JSON columns by identity.
|
||||
document.document_metadata = metadata
|
||||
await session.commit()
|
||||
except Exception:
|
||||
logger.warning("Could not record seeded paths", exc_info=True)
|
||||
await session.rollback()
|
||||
|
|
@ -1,69 +0,0 @@
|
|||
"""Knowledge-store enablement: process-wide config and per-workspace flip state.
|
||||
|
||||
The global ``KNOWLEDGE_STORE_ENABLED`` env is the master kill switch; each
|
||||
workspace flips individually via ``workspaces.knowledge_store_enabled`` after
|
||||
its migration seed passes parity. A workspace is git-native only when both
|
||||
are on.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class KnowledgeStoreSettings:
|
||||
"""Resolved knowledge-store configuration for the current process."""
|
||||
|
||||
enabled: bool
|
||||
root: str
|
||||
|
||||
|
||||
def load_knowledge_store_settings() -> KnowledgeStoreSettings:
|
||||
"""Resolve knowledge-store settings from the central ``Config`` singleton."""
|
||||
from app.config import config
|
||||
|
||||
return KnowledgeStoreSettings(
|
||||
enabled=config.KNOWLEDGE_STORE_ENABLED,
|
||||
root=config.KNOWLEDGE_STORE_ROOT,
|
||||
)
|
||||
|
||||
|
||||
_FLAG_TTL_SECONDS = 30.0
|
||||
# ponytail: in-process TTL cache, so a flip propagates within 30s per process;
|
||||
# pub/sub invalidation is the upgrade if that window ever matters.
|
||||
_flag_cache: dict[int, tuple[bool, float]] = {}
|
||||
|
||||
|
||||
async def knowledge_store_enabled_for(workspace_id: int) -> bool:
|
||||
"""Whether ``workspace_id`` is git-native right now.
|
||||
|
||||
True only when the global master switch and the workspace's own flip
|
||||
flag are both on. The workspace flag is cached per process; the global
|
||||
switch is read live, so killing it takes effect immediately.
|
||||
"""
|
||||
if not load_knowledge_store_settings().enabled:
|
||||
return False
|
||||
cached = _flag_cache.get(workspace_id)
|
||||
if cached and cached[1] > time.monotonic():
|
||||
return cached[0]
|
||||
enabled = await _read_workspace_flag(workspace_id)
|
||||
_flag_cache[workspace_id] = (enabled, time.monotonic() + _FLAG_TTL_SECONDS)
|
||||
return enabled
|
||||
|
||||
|
||||
async def _read_workspace_flag(workspace_id: int) -> bool:
|
||||
"""``workspaces.knowledge_store_enabled`` for one row; False if absent."""
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.db import Workspace, async_session_maker
|
||||
|
||||
async with async_session_maker() as session:
|
||||
return bool(
|
||||
await session.scalar(
|
||||
select(Workspace.knowledge_store_enabled).where(
|
||||
Workspace.id == workspace_id
|
||||
)
|
||||
)
|
||||
)
|
||||
|
|
@ -1,120 +0,0 @@
|
|||
"""Public, engine-agnostic API for a workspace's versioned content."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from app.knowledge_store.engines.base import (
|
||||
Change,
|
||||
Revision,
|
||||
TrackedPath,
|
||||
VersionedContentEngine,
|
||||
WorkingCopy,
|
||||
)
|
||||
from app.knowledge_store.engines.git import GitContentEngine
|
||||
from app.knowledge_store.store_path import (
|
||||
workspace_store_path,
|
||||
workspace_working_copies_path,
|
||||
)
|
||||
from app.knowledge_store.transaction import Transaction
|
||||
from app.knowledge_store.write_lock import workspace_write_lock
|
||||
|
||||
|
||||
class KnowledgeStore:
|
||||
"""Versioned content history for one workspace."""
|
||||
|
||||
def __init__(self, workspace_id: int | str, engine: VersionedContentEngine) -> None:
|
||||
self._workspace_id = workspace_id
|
||||
self._engine = engine
|
||||
|
||||
@classmethod
|
||||
def for_workspace(cls, workspace_id: int | str) -> KnowledgeStore:
|
||||
"""Build a workspace's store; the only place it binds to a concrete engine."""
|
||||
engine = GitContentEngine(
|
||||
workspace_store_path(workspace_id),
|
||||
workspace_working_copies_path(workspace_id),
|
||||
)
|
||||
return cls(workspace_id, engine)
|
||||
|
||||
@asynccontextmanager
|
||||
async def transaction(
|
||||
self, *, message: str, author: str, committer: str | None = None
|
||||
):
|
||||
"""Atomic unit of work: verbs staged in the scope become one revision
|
||||
on clean exit; an exception records nothing.
|
||||
|
||||
``author`` is whose content change this is; ``committer`` (default
|
||||
``author``) is who recorded it — the agent identity for agent turns."""
|
||||
tx = Transaction()
|
||||
yield tx
|
||||
async with workspace_write_lock(self._workspace_id):
|
||||
tx.revision = await asyncio.to_thread(
|
||||
self._record_revision, tx, message, author, committer
|
||||
)
|
||||
|
||||
async def read_as_of(self, revision: str, path: str) -> bytes:
|
||||
"""Bytes of ``path`` as of ``revision``."""
|
||||
return await asyncio.to_thread(self._engine.read_as_of, revision, path)
|
||||
|
||||
async def list_revisions(
|
||||
self, *, path: str | None = None, limit: int | None = None
|
||||
) -> list[Revision]:
|
||||
"""Revisions newest-first, optionally scoped to a single ``path``."""
|
||||
return await asyncio.to_thread(
|
||||
self._engine.list_revisions, path=path, limit=limit
|
||||
)
|
||||
|
||||
async def list_changes(
|
||||
self, revision: str, *, since: str | None = None
|
||||
) -> list[Change]:
|
||||
"""What ``revision`` changed, against its parent or against ``since``."""
|
||||
return await asyncio.to_thread(self._engine.list_changes, revision, since=since)
|
||||
|
||||
async def list_paths(self, revision: str) -> list[TrackedPath]:
|
||||
"""Every path stored at ``revision``, with its content address."""
|
||||
return await asyncio.to_thread(self._engine.list_paths, revision)
|
||||
|
||||
async def get_current_revision(self) -> str | None:
|
||||
"""Id of the workspace's current revision (a whole-workspace snapshot),
|
||||
or ``None`` when the store is empty."""
|
||||
return await asyncio.to_thread(self._engine.get_current_revision)
|
||||
|
||||
async def open_working_copy(self, copy_id: str) -> WorkingCopy:
|
||||
"""Private on-disk copy of the current content; reopens an existing one."""
|
||||
return await asyncio.to_thread(self._engine.open_working_copy, copy_id)
|
||||
|
||||
async def diff_working_copy(
|
||||
self, copy_id: str
|
||||
) -> tuple[dict[str, bytes], list[str]]:
|
||||
"""Net changes in ``copy_id``'s copy since its base, as ``(writes, removes)``."""
|
||||
return await asyncio.to_thread(self._engine.diff_working_copy, copy_id)
|
||||
|
||||
async def discard_working_copy(self, copy_id: str) -> None:
|
||||
"""Delete ``copy_id``'s working copy; a no-op if absent."""
|
||||
await asyncio.to_thread(self._engine.discard_working_copy, copy_id)
|
||||
|
||||
async def prune_working_copies(self, *, older_than_seconds: float) -> list[str]:
|
||||
"""Delete abandoned working copies; returns the pruned ids."""
|
||||
return await asyncio.to_thread(
|
||||
lambda: self._engine.prune_working_copies(
|
||||
older_than_seconds=older_than_seconds
|
||||
)
|
||||
)
|
||||
|
||||
def compute_content_id(self, data: bytes) -> str:
|
||||
"""Content address for ``data`` (no I/O)."""
|
||||
return self._engine.compute_content_id(data)
|
||||
|
||||
def _record_revision(
|
||||
self, tx: Transaction, message: str, author: str, committer: str | None
|
||||
) -> str | None:
|
||||
"""Resolve the transaction into one change set and record it."""
|
||||
writes, removes = tx.resolve(self._engine.read)
|
||||
return self._engine.record(
|
||||
writes=writes,
|
||||
removes=removes,
|
||||
message=message,
|
||||
author=author,
|
||||
committer=committer,
|
||||
)
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
"""On-disk location of a workspace's versioned store (single owner of the layout)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from app.knowledge_store.settings import load_knowledge_store_settings
|
||||
|
||||
|
||||
def workspace_store_path(workspace_id: int | str) -> Path:
|
||||
"""Absolute directory holding a single workspace's versioned history."""
|
||||
root = load_knowledge_store_settings().root
|
||||
return Path(root) / str(workspace_id)
|
||||
|
||||
|
||||
def working_copies_root() -> Path:
|
||||
"""Absolute directory holding every workspace's working copies."""
|
||||
root = load_knowledge_store_settings().root
|
||||
return Path(root) / ".working_copies"
|
||||
|
||||
|
||||
def workspace_working_copies_path(workspace_id: int | str) -> Path:
|
||||
"""Absolute directory holding a workspace's private working copies."""
|
||||
return working_copies_root() / str(workspace_id)
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
"""One atomic unit of work; its staged verbs become a single revision."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
|
||||
|
||||
class Transaction:
|
||||
"""An open unit of work: stage intent verbs, recorded atomically on exit."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._writes: dict[str, bytes] = {}
|
||||
self._removes: list[str] = []
|
||||
self._moves: list[tuple[str, str]] = []
|
||||
#: Resulting revision id, set on scope exit (``None`` when nothing changed).
|
||||
self.revision: str | None = None
|
||||
|
||||
def write(self, path: str, content: bytes) -> None:
|
||||
"""Create or replace ``path``."""
|
||||
self._writes[path] = content
|
||||
if path in self._removes:
|
||||
self._removes.remove(path)
|
||||
|
||||
def remove(self, path: str) -> None:
|
||||
"""Delete ``path``."""
|
||||
self._writes.pop(path, None)
|
||||
if path not in self._removes:
|
||||
self._removes.append(path)
|
||||
|
||||
def move(self, src: str, dst: str) -> None:
|
||||
"""Relocate ``src`` to ``dst``."""
|
||||
self._moves.append((src, dst))
|
||||
|
||||
def resolve(
|
||||
self, read_current: Callable[[str], bytes | None]
|
||||
) -> tuple[dict[str, bytes], list[str]]:
|
||||
"""Resolve the staged verbs (moves included) into concrete writes/removes."""
|
||||
writes = dict(self._writes)
|
||||
removes = list(self._removes)
|
||||
for src, dst in self._moves:
|
||||
content = writes.pop(src, None)
|
||||
if content is None:
|
||||
content = read_current(src)
|
||||
if content is None:
|
||||
raise FileNotFoundError(f"cannot move missing path: {src}")
|
||||
writes[dst] = content
|
||||
removes.append(src)
|
||||
return writes, removes
|
||||
|
|
@ -1,110 +0,0 @@
|
|||
"""Cross-process per-workspace locks.
|
||||
|
||||
A write never proceeds unserialized: if the lock cannot be acquired (contention
|
||||
timeout, or Redis unreachable), the caller fails instead of racing. A hold that
|
||||
outlives its TTL fails just as loudly — exclusivity was lost, never silently.
|
||||
|
||||
Indexing takes a *separate* lock. It embeds, so it runs for far longer than a
|
||||
commit — sharing the write lock would stall agent writes behind embedding calls,
|
||||
and sizing one TTL for both would either wedge writes or expire mid-rebuild.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import asynccontextmanager, suppress
|
||||
|
||||
import redis.asyncio as redis
|
||||
from redis.exceptions import LockError, LockNotOwnedError
|
||||
|
||||
from app.config import config
|
||||
|
||||
# Auto-expiry so a crashed writer can't wedge a workspace; must outlast a write.
|
||||
LOCK_TTL_SECONDS = 30.0
|
||||
# How long a contender waits before giving up.
|
||||
LOCK_WAIT_SECONDS = 10.0
|
||||
|
||||
# Indexing a whole workspace embeds every document, so its ceiling is minutes.
|
||||
INDEX_LOCK_TTL_SECONDS = 1800.0
|
||||
# A contender gives up quickly: the holder converges to the current revision
|
||||
# anyway, and the drift sweep re-drives anything it missed.
|
||||
INDEX_LOCK_WAIT_SECONDS = 5.0
|
||||
|
||||
|
||||
class KnowledgeStoreLockError(RuntimeError):
|
||||
"""A workspace lock could not be acquired, or a hold expired mid-block."""
|
||||
|
||||
|
||||
def _lock_key(workspace_id: int | str, purpose: str) -> str:
|
||||
return f"knowledge_store:{purpose}:{workspace_id}"
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _workspace_lock(
|
||||
workspace_id: int | str, *, purpose: str, ttl: float, wait: float
|
||||
):
|
||||
# The client lives and dies with the block rather than being cached: celery
|
||||
# runs every task on a fresh event loop, and a pooled connection bound to a
|
||||
# closed one fails on the next task. It failed *inside* acquire, after redis
|
||||
# had set the key but before the reply was read — leaving the lock held by
|
||||
# nobody for its whole TTL. A connection per lock is cheap next to the write
|
||||
# it guards.
|
||||
client = redis.from_url(config.REDIS_APP_URL, decode_responses=True)
|
||||
try:
|
||||
lock = client.lock(
|
||||
_lock_key(workspace_id, purpose),
|
||||
timeout=ttl,
|
||||
blocking=True,
|
||||
blocking_timeout=wait,
|
||||
)
|
||||
if not await lock.acquire():
|
||||
raise KnowledgeStoreLockError(
|
||||
f"Could not acquire {purpose} for workspace {workspace_id} "
|
||||
f"within {wait}s"
|
||||
)
|
||||
try:
|
||||
yield
|
||||
except BaseException:
|
||||
# The block itself failed; a lost hold must not mask that error.
|
||||
with suppress(LockError):
|
||||
await lock.release()
|
||||
raise
|
||||
try:
|
||||
await lock.release()
|
||||
except LockNotOwnedError:
|
||||
# The hold outlived the TTL: the work landed, but its tail ran
|
||||
# without exclusivity. Fail loudly instead of hiding the race.
|
||||
raise KnowledgeStoreLockError(
|
||||
f"{purpose} for workspace {workspace_id} expired mid-block "
|
||||
f"(hold exceeded the {ttl}s TTL)"
|
||||
) from None
|
||||
finally:
|
||||
with suppress(Exception):
|
||||
await client.aclose()
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def workspace_write_lock(workspace_id: int | str):
|
||||
"""Hold ``workspace_id``'s single-writer lock for the block."""
|
||||
async with _workspace_lock(
|
||||
workspace_id,
|
||||
purpose="write_lock",
|
||||
ttl=LOCK_TTL_SECONDS,
|
||||
wait=LOCK_WAIT_SECONDS,
|
||||
):
|
||||
yield
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def workspace_index_lock(workspace_id: int | str):
|
||||
"""Hold ``workspace_id``'s single-indexer lock for the block.
|
||||
|
||||
``ponytail:`` ceiling — a rebuild that outruns the TTL can be joined by a
|
||||
second builder; upgrade path is a ``lock.extend()`` heartbeat while indexing.
|
||||
"""
|
||||
async with _workspace_lock(
|
||||
workspace_id,
|
||||
purpose="index_lock",
|
||||
ttl=INDEX_LOCK_TTL_SECONDS,
|
||||
wait=INDEX_LOCK_WAIT_SECONDS,
|
||||
):
|
||||
yield
|
||||
|
|
@ -490,22 +490,6 @@ def _gateway_webhook_parse_errors():
|
|||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _knowledge_store_record_outcome():
|
||||
return _get_meter().create_counter(
|
||||
"surfsense.knowledge_store.record.outcome",
|
||||
description="Count of knowledge-store recording outcomes per write flow.",
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _knowledge_store_drift_checks():
|
||||
return _get_meter().create_counter(
|
||||
"surfsense.knowledge_store.drift.check",
|
||||
description="Count of scheduled knowledge-store parity checks per outcome.",
|
||||
)
|
||||
|
||||
|
||||
def record_model_call_duration(
|
||||
duration_ms: float, *, model: str | None, provider: str | None
|
||||
) -> None:
|
||||
|
|
@ -888,33 +872,6 @@ def record_gateway_webhook_parse_error() -> None:
|
|||
_add(_gateway_webhook_parse_errors(), 1, {})
|
||||
|
||||
|
||||
def record_knowledge_store_record_outcome(
|
||||
*, flow: str, status: str, error_category: str | None = None
|
||||
) -> None:
|
||||
"""Record one knowledge-store recording attempt.
|
||||
|
||||
``flow`` names the write path (``editor_save``, ``sync_batch``,
|
||||
``turn_commit``); ``status`` is ``recorded``, ``noop``, or ``failed``.
|
||||
A non-zero ``failed`` rate means git is drifting behind Postgres.
|
||||
"""
|
||||
_add(
|
||||
_knowledge_store_record_outcome(),
|
||||
1,
|
||||
_attrs_with_optional_error_category(
|
||||
{"flow": flow, "status": status}, error_category
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def record_knowledge_store_drift_check(*, workspace_id: int, status: str) -> None:
|
||||
"""Record one scheduled parity check. ``status`` is ``ok``, ``drift``, or ``error``."""
|
||||
_add(
|
||||
_knowledge_store_drift_checks(),
|
||||
1,
|
||||
{"workspace.id": workspace_id, "status": status},
|
||||
)
|
||||
|
||||
|
||||
def _runtime_snapshot_value(key: str, transform: Any = None) -> list[Any]:
|
||||
from opentelemetry.metrics import Observation
|
||||
|
||||
|
|
@ -1022,8 +979,6 @@ __all__ = [
|
|||
"record_indexing_document_outcome",
|
||||
"record_interrupt",
|
||||
"record_kb_search_duration",
|
||||
"record_knowledge_store_drift_check",
|
||||
"record_knowledge_store_record_outcome",
|
||||
"record_model_call_duration",
|
||||
"record_model_token_usage",
|
||||
"record_perf_elapsed",
|
||||
|
|
|
|||
|
|
@ -21,7 +21,6 @@ from app.db import (
|
|||
WorkspaceMembership,
|
||||
get_async_session,
|
||||
)
|
||||
from app.knowledge_store.settings import knowledge_store_enabled_for
|
||||
from app.schemas import (
|
||||
ChunkRead,
|
||||
DocumentRead,
|
||||
|
|
@ -1467,6 +1466,7 @@ async def list_document_versions(
|
|||
session: AsyncSession = Depends(get_async_session),
|
||||
auth: AuthContext = Depends(get_auth_context),
|
||||
):
|
||||
user = auth.user
|
||||
"""List all versions for a document, ordered by version_number descending."""
|
||||
document = (
|
||||
await session.execute(select(Document).where(Document.id == document_id))
|
||||
|
|
@ -1475,7 +1475,7 @@ async def list_document_versions(
|
|||
raise HTTPException(status_code=404, detail="Document not found")
|
||||
|
||||
await check_permission(
|
||||
session, auth, document.workspace_id, Permission.DOCUMENTS_READ.value
|
||||
session, user, document.workspace_id, Permission.DOCUMENTS_READ.value
|
||||
)
|
||||
|
||||
versions = (
|
||||
|
|
@ -1508,6 +1508,7 @@ async def get_document_version(
|
|||
session: AsyncSession = Depends(get_async_session),
|
||||
auth: AuthContext = Depends(get_auth_context),
|
||||
):
|
||||
user = auth.user
|
||||
"""Get full version content including source_markdown."""
|
||||
document = (
|
||||
await session.execute(select(Document).where(Document.id == document_id))
|
||||
|
|
@ -1516,7 +1517,7 @@ async def get_document_version(
|
|||
raise HTTPException(status_code=404, detail="Document not found")
|
||||
|
||||
await check_permission(
|
||||
session, auth, document.workspace_id, Permission.DOCUMENTS_READ.value
|
||||
session, user, document.workspace_id, Permission.DOCUMENTS_READ.value
|
||||
)
|
||||
|
||||
version = (
|
||||
|
|
@ -1555,22 +1556,9 @@ async def restore_document_version(
|
|||
raise HTTPException(status_code=404, detail="Document not found")
|
||||
|
||||
await check_permission(
|
||||
session, auth, document.workspace_id, Permission.DOCUMENTS_UPDATE.value
|
||||
session, user, document.workspace_id, Permission.DOCUMENTS_UPDATE.value
|
||||
)
|
||||
|
||||
if await knowledge_store_enabled_for(document.workspace_id):
|
||||
# Restore rewrites source_markdown and title without recording a
|
||||
# revision, so git — the source of truth — would still hold the newer
|
||||
# content: search would keep serving it and the next reindex would revert
|
||||
# the restore outright. History is `git revert` for these workspaces.
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=(
|
||||
"Version restore is unavailable for git-backed workspaces; "
|
||||
"revert the change in document history instead."
|
||||
),
|
||||
)
|
||||
|
||||
version = (
|
||||
await session.execute(
|
||||
select(DocumentVersion).where(
|
||||
|
|
|
|||
|
|
@ -27,7 +27,6 @@ from app.routes.reports_routes import (
|
|||
_normalize_latex_delimiters,
|
||||
_strip_wrapping_code_fences,
|
||||
)
|
||||
from app.services.document_revision_recorder import record_saved_document
|
||||
from app.templates.export_helpers import (
|
||||
get_html_css_path,
|
||||
get_reference_docx_path,
|
||||
|
|
@ -285,10 +284,9 @@ async def save_document(
|
|||
raise HTTPException(status_code=400, detail="source_markdown must be a string")
|
||||
|
||||
# For NOTE type, extract title from first heading line if present
|
||||
provided_title = data.get("title")
|
||||
if document.document_type == DocumentType.NOTE:
|
||||
# If the frontend sends a title, use it; otherwise extract from markdown
|
||||
new_title = provided_title
|
||||
new_title = data.get("title")
|
||||
if not new_title:
|
||||
# Extract title from the first line of markdown (# Heading)
|
||||
for line in source_markdown.split("\n"):
|
||||
|
|
@ -313,18 +311,6 @@ async def save_document(
|
|||
|
||||
await session.commit()
|
||||
|
||||
await record_saved_document(
|
||||
session,
|
||||
workspace_id=workspace_id,
|
||||
doc_id=document.id,
|
||||
title=document.title,
|
||||
folder_id=document.folder_id,
|
||||
markdown=source_markdown,
|
||||
author_user_id=str(user.id),
|
||||
# A title read back off the heading above is not a rename request.
|
||||
title_is_explicit=bool(provided_title),
|
||||
)
|
||||
|
||||
# Queue reindex task
|
||||
reindex_document_task.delay(document_id, str(user.id))
|
||||
|
||||
|
|
|
|||
|
|
@ -1,215 +0,0 @@
|
|||
"""Direct-caller adapter: document content changes become knowledge-store revisions.
|
||||
|
||||
Editor saves, upload-extracted markdown, and connector sync batches share this
|
||||
path — the same single write path agent turns use — behind the per-workspace
|
||||
knowledge-store flag.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.agents.chat.runtime.path_resolver import (
|
||||
PATH_MARKER,
|
||||
build_path_index,
|
||||
doc_to_virtual_path,
|
||||
to_store_path,
|
||||
virtual_path_of,
|
||||
)
|
||||
from app.db import Document
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.knowledge_store.identities import user_identity
|
||||
from app.knowledge_store.index.queue import enqueue_index
|
||||
from app.knowledge_store.settings import (
|
||||
knowledge_store_enabled_for,
|
||||
load_knowledge_store_settings,
|
||||
)
|
||||
from app.observability import metrics
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.db import Document
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def record_markdown_files(
|
||||
*,
|
||||
workspace_id: int | str,
|
||||
files: Mapping[str, str],
|
||||
message: str,
|
||||
author_user_id: str | None,
|
||||
removes: Sequence[str] = (),
|
||||
) -> str | None:
|
||||
"""Record ``files`` (store path → markdown) as one revision.
|
||||
|
||||
``removes`` drops paths in the same revision as the writes — otherwise a
|
||||
retitle would leave one document behind as two files.
|
||||
|
||||
``None`` when the store is disabled, the batch is empty, or nothing
|
||||
actually changed (identical content is a no-op by construction).
|
||||
"""
|
||||
if (not files and not removes) or not load_knowledge_store_settings().enabled:
|
||||
return None
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
async with store.transaction(
|
||||
message=message, author=user_identity(author_user_id)
|
||||
) as tx:
|
||||
for path, markdown in files.items():
|
||||
tx.write(path, markdown.encode())
|
||||
for path in removes:
|
||||
tx.remove(path)
|
||||
if tx.revision is not None:
|
||||
enqueue_index(workspace_id)
|
||||
return tx.revision
|
||||
|
||||
|
||||
async def record_saved_document(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
workspace_id: int,
|
||||
doc_id: int,
|
||||
title: str,
|
||||
folder_id: int | None,
|
||||
markdown: str,
|
||||
author_user_id: str | None,
|
||||
title_is_explicit: bool = False,
|
||||
) -> str | None:
|
||||
"""Resolve one document's canonical store path and record the save.
|
||||
|
||||
The recorded path is remembered on the row (``document_metadata``), so the
|
||||
next save knows where the document used to live and can drop that file
|
||||
when a retitle moves it. The marker is written only when a revision was
|
||||
actually recorded: a marker without a file would make the row look
|
||||
indexer-owned and a later rebuild would prune it.
|
||||
|
||||
``title_is_explicit`` means someone chose the title, so the file follows it.
|
||||
A note's title is otherwise re-read from its first heading on every save,
|
||||
and letting that place the file would rename whatever the agent named — for
|
||||
a name the caller never asked to change.
|
||||
|
||||
Never raises: while the store coexists with the Postgres write path
|
||||
(until the Phase 5 cut), a recording failure must not fail the save
|
||||
that already committed — it is logged instead.
|
||||
"""
|
||||
if not await knowledge_store_enabled_for(workspace_id):
|
||||
return None
|
||||
try:
|
||||
index = await build_path_index(session, workspace_id)
|
||||
document = await session.get(Document, doc_id)
|
||||
metadata = document.document_metadata if document else None
|
||||
previous = (metadata or {}).get(PATH_MARKER)
|
||||
virtual_path = (
|
||||
doc_to_virtual_path(
|
||||
doc_id=doc_id, title=title, folder_id=folder_id, index=index
|
||||
)
|
||||
if title_is_explicit
|
||||
else virtual_path_of(
|
||||
metadata=metadata,
|
||||
doc_id=doc_id,
|
||||
title=title,
|
||||
folder_id=folder_id,
|
||||
index=index,
|
||||
)
|
||||
)
|
||||
filename = virtual_path.rsplit("/", 1)[-1]
|
||||
stale = _stale_store_path(previous, virtual_path)
|
||||
revision = await record_markdown_files(
|
||||
workspace_id=workspace_id,
|
||||
files={to_store_path(virtual_path): markdown},
|
||||
message=f"docs: save {filename}",
|
||||
author_user_id=author_user_id,
|
||||
removes=[stale] if stale else (),
|
||||
)
|
||||
if revision is not None and document is not None and previous != virtual_path:
|
||||
document.document_metadata = {
|
||||
**(document.document_metadata or {}),
|
||||
PATH_MARKER: virtual_path,
|
||||
}
|
||||
# Safe to commit here: the recorder runs at the point of
|
||||
# durability, after the save's own commit, so nothing else is
|
||||
# pending on this session.
|
||||
await session.commit()
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Knowledge store recording failed for document %s in workspace %s",
|
||||
doc_id,
|
||||
workspace_id,
|
||||
exc_info=True,
|
||||
)
|
||||
metrics.record_knowledge_store_record_outcome(
|
||||
flow="editor_save",
|
||||
status="failed",
|
||||
error_category=metrics.categorize_exception(exc),
|
||||
)
|
||||
return None
|
||||
metrics.record_knowledge_store_record_outcome(
|
||||
flow="editor_save", status="recorded" if revision else "noop"
|
||||
)
|
||||
return revision
|
||||
|
||||
|
||||
async def record_prepared_documents(
|
||||
session: AsyncSession, documents: Sequence[Document]
|
||||
) -> str | None:
|
||||
"""Record a sync batch's accepted markdown as one revision.
|
||||
|
||||
Called after ``prepare_for_indexing`` commits — the moment content becomes
|
||||
durable — so chunking/embedding failures can never block the record.
|
||||
Never raises, for the same coexistence reason as ``record_saved_document``.
|
||||
"""
|
||||
if not documents:
|
||||
return None
|
||||
workspace_id = documents[0].workspace_id
|
||||
if not await knowledge_store_enabled_for(workspace_id):
|
||||
return None
|
||||
try:
|
||||
index = await build_path_index(session, workspace_id)
|
||||
files: dict[str, str] = {}
|
||||
for doc in documents:
|
||||
if not doc.source_markdown:
|
||||
continue
|
||||
virtual_path = doc_to_virtual_path(
|
||||
doc_id=doc.id, title=doc.title, folder_id=doc.folder_id, index=index
|
||||
)
|
||||
files[to_store_path(virtual_path)] = doc.source_markdown
|
||||
revision = await record_markdown_files(
|
||||
workspace_id=workspace_id,
|
||||
files=files,
|
||||
message=f"sync: index {len(files)} document(s)",
|
||||
author_user_id=(
|
||||
str(documents[0].created_by_id)
|
||||
if documents[0].created_by_id is not None
|
||||
else None
|
||||
),
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Knowledge store recording failed for a sync batch in workspace %s",
|
||||
workspace_id,
|
||||
exc_info=True,
|
||||
)
|
||||
metrics.record_knowledge_store_record_outcome(
|
||||
flow="sync_batch",
|
||||
status="failed",
|
||||
error_category=metrics.categorize_exception(exc),
|
||||
)
|
||||
return None
|
||||
metrics.record_knowledge_store_record_outcome(
|
||||
flow="sync_batch", status="recorded" if revision else "noop"
|
||||
)
|
||||
return revision
|
||||
|
||||
|
||||
def _stale_store_path(previous: str | None, current: str) -> str | None:
|
||||
"""Store path the document is moving away from, if it is moving at all."""
|
||||
if not previous or previous == current:
|
||||
return None
|
||||
try:
|
||||
return to_store_path(previous)
|
||||
except ValueError:
|
||||
# A marker from outside the /documents namespace is not ours to drop.
|
||||
return None
|
||||
|
|
@ -9,7 +9,6 @@ from sqlalchemy.orm import selectinload
|
|||
from app.celery_app import celery_app
|
||||
from app.db import Document
|
||||
from app.indexing_pipeline.adapters.file_upload_adapter import UploadDocumentAdapter
|
||||
from app.knowledge_store.settings import knowledge_store_enabled_for
|
||||
from app.services.task_logging_service import TaskLoggingService
|
||||
from app.tasks.celery_tasks import get_celery_session_maker, run_async_celery_task
|
||||
|
||||
|
|
@ -42,20 +41,6 @@ async def _reindex_document(document_id: int, user_id: str):
|
|||
logger.error(f"Document {document_id} not found")
|
||||
return
|
||||
|
||||
if await knowledge_store_enabled_for(document.workspace_id):
|
||||
# The store indexer owns this document's chunks. Both writers would
|
||||
# otherwise reconcile the same rows from different sources — this one
|
||||
# from Postgres `source_markdown` with the editor's first-heading
|
||||
# title, the indexer from git with the filename stem — so the title
|
||||
# would flip on every save. Guarded here rather than at the two call
|
||||
# sites so neither can be missed.
|
||||
logger.info(
|
||||
"Skipping editor reindex of document %s; "
|
||||
"the knowledge-store indexer owns its chunks",
|
||||
document_id,
|
||||
)
|
||||
return
|
||||
|
||||
task_logger = TaskLoggingService(session, document.workspace_id)
|
||||
|
||||
log_entry = await task_logger.log_task_start(
|
||||
|
|
|
|||
|
|
@ -1,5 +0,0 @@
|
|||
"""Celery entry points for the knowledge store.
|
||||
|
||||
Task modules are discovered through ``celery_app``'s ``include`` list, so
|
||||
nothing is re-exported here.
|
||||
"""
|
||||
|
|
@ -1,103 +0,0 @@
|
|||
"""Celery task detecting — and repairing — Postgres↔git drift on flipped workspaces.
|
||||
|
||||
The always-on version of the fleet runner's dry run: for every workspace with
|
||||
``knowledge_store_enabled``, compare the store's head against Postgres by
|
||||
content address and emit one ``knowledge_store.drift.check`` data point.
|
||||
An alert on ``status != ok`` replaces reading JSONL reports by hand.
|
||||
|
||||
The hourly sweep converges the drift it can see, which is only git running
|
||||
ahead of the stamp; drift on the Postgres side is invisible to it, because both
|
||||
sides of its comparison are git revisions. This check sees that drift, so it
|
||||
also closes it — leaving the fix as a runbook step would make repair depend on
|
||||
someone noticing the alert.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.celery_app import celery_app
|
||||
from app.knowledge_store.migrate import MigrationReport, migrate_workspace
|
||||
from app.knowledge_store.settings import load_knowledge_store_settings
|
||||
from app.observability import metrics
|
||||
from app.tasks.celery_tasks.knowledge_store.index_tasks import reindex_knowledge_store
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Repairs enqueued per run. Drift should be rare, so a run wanting more than a
|
||||
#: handful is a systemic problem, and fanning out whole-tree converges would
|
||||
#: compound it rather than fix it; the rest wait for the next run.
|
||||
#:
|
||||
#: ponytail: drift ``index_tree`` cannot fix — a Postgres row carrying no path
|
||||
#: marker and having no file in the tree, i.e. some writer bypassing git — costs
|
||||
#: one rebuild per run until a human intervenes. The alarm persists throughout,
|
||||
#: so it stays visible; the upgrade path is a per-workspace attempt count to
|
||||
#: back off after a repair that changed nothing.
|
||||
REPAIR_ENQUEUE_CAP = 10
|
||||
|
||||
|
||||
@celery_app.task(name="check_knowledge_store_drift")
|
||||
def check_knowledge_store_drift() -> dict[str, int]:
|
||||
"""Return status counts, e.g. ``{"ok": 12, "drift": 1}``."""
|
||||
if not load_knowledge_store_settings().enabled:
|
||||
return {}
|
||||
return asyncio.run(_check_flipped_workspaces())
|
||||
|
||||
|
||||
async def _check_flipped_workspaces() -> dict[str, int]:
|
||||
from app.db import Workspace, async_session_maker
|
||||
|
||||
async with async_session_maker() as session:
|
||||
workspace_ids = (
|
||||
(
|
||||
await session.execute(
|
||||
select(Workspace.id).where(
|
||||
Workspace.knowledge_store_enabled.is_(True)
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
|
||||
counts: dict[str, int] = {}
|
||||
repairs = 0
|
||||
for workspace_id in workspace_ids:
|
||||
# Fresh session per workspace, like the fleet runner: one workspace's
|
||||
# failure must not poison the next check.
|
||||
async with async_session_maker() as session:
|
||||
report = await migrate_workspace(session, workspace_id, dry_run=True)
|
||||
status = _status(report)
|
||||
counts[status] = counts.get(status, 0) + 1
|
||||
metrics.record_knowledge_store_drift_check(
|
||||
workspace_id=workspace_id, status=status
|
||||
)
|
||||
if status != "ok":
|
||||
logger.warning(
|
||||
"Knowledge store drift check for workspace %s: %s "
|
||||
"(missing=%d extra=%d mismatched=%d error=%s)",
|
||||
workspace_id,
|
||||
status,
|
||||
len(report.missing),
|
||||
len(report.extra),
|
||||
len(report.mismatched),
|
||||
report.error,
|
||||
)
|
||||
if status == "drift" and repairs < REPAIR_ENQUEUE_CAP:
|
||||
# git is the truth, so the whole-tree converge is the repair: it
|
||||
# upserts rows for paths Postgres lacks, overwrites content that
|
||||
# disagrees, and prunes marked rows whose file is gone. `error` is
|
||||
# deliberately excluded — a store the check could not read will not
|
||||
# be fixed by indexing it harder.
|
||||
reindex_knowledge_store.delay(workspace_id)
|
||||
repairs += 1
|
||||
return counts
|
||||
|
||||
|
||||
def _status(report: MigrationReport) -> str:
|
||||
if report.error:
|
||||
return "error"
|
||||
return "ok" if report.ok else "drift"
|
||||
|
|
@ -1,140 +0,0 @@
|
|||
"""Celery tasks that keep the derived index converged with the store.
|
||||
|
||||
One task per store write (enqueued by the writers), one rebuild task, and an
|
||||
hourly drift sweep that re-drives anything the first two lost.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.celery_app import celery_app
|
||||
from app.db import Workspace
|
||||
from app.knowledge_store.index.converge import index_changes, index_tree
|
||||
from app.knowledge_store.settings import (
|
||||
knowledge_store_enabled_for,
|
||||
load_knowledge_store_settings,
|
||||
)
|
||||
from app.knowledge_store.store import KnowledgeStore
|
||||
from app.knowledge_store.write_lock import KnowledgeStoreLockError
|
||||
from app.tasks.celery_tasks import get_celery_session_maker, run_async_celery_task
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Enqueues per sweep. The drift check itself is one HEAD read, but the tasks it
|
||||
#: fans out each embed, so the cap is what keeps a fleet-wide backfill (many
|
||||
#: workspaces flipped at once) from burying user-facing work.
|
||||
SWEEP_ENQUEUE_CAP = 100
|
||||
|
||||
#: A held lock means a converge is in flight; if it read HEAD before this save
|
||||
#: landed, only a later run picks the save up — so retry rather than leave the
|
||||
#: save stale until the hourly sweep. A redundant retry no-ops on the stamp.
|
||||
LOCK_RETRY_DELAY_SECONDS = 30
|
||||
#: ponytail: 10 x 30s rides out one full rebuild; anything longer-lived falls
|
||||
#: back to the sweep. Upgrade path is delay scaled to the holder's lock TTL.
|
||||
LOCK_RETRY_LIMIT = 10
|
||||
|
||||
|
||||
@celery_app.task(
|
||||
name="index_knowledge_store_revision", bind=True, max_retries=LOCK_RETRY_LIMIT
|
||||
)
|
||||
def index_knowledge_store_revision(self, workspace_id: int) -> int:
|
||||
"""Fold the workspace store's current revision into the index."""
|
||||
if not load_knowledge_store_settings().enabled:
|
||||
return 0
|
||||
try:
|
||||
return run_async_celery_task(lambda: _index(workspace_id, full=False))
|
||||
except KnowledgeStoreLockError as exc:
|
||||
raise self.retry(countdown=LOCK_RETRY_DELAY_SECONDS, exc=exc) from exc
|
||||
|
||||
|
||||
@celery_app.task(name="reindex_knowledge_store")
|
||||
def reindex_knowledge_store(workspace_id: int) -> int:
|
||||
"""Rebuild a workspace's whole index from its current tree."""
|
||||
if not load_knowledge_store_settings().enabled:
|
||||
return 0
|
||||
return run_async_celery_task(lambda: _index(workspace_id, full=True))
|
||||
|
||||
|
||||
@celery_app.task(name="reindex_drifted_workspaces")
|
||||
def reindex_drifted_workspaces() -> int:
|
||||
"""Enqueue indexing for flipped workspaces whose stamp trails their store."""
|
||||
if not load_knowledge_store_settings().enabled:
|
||||
return 0
|
||||
return run_async_celery_task(_sweep)
|
||||
|
||||
|
||||
async def _index(workspace_id: int, *, full: bool) -> int:
|
||||
"""Run one convergence; return how many documents it indexed.
|
||||
|
||||
The per-workspace flag is re-checked here, at the worker: a task can sit
|
||||
in the queue across an unflip, and indexing an unflipped workspace would
|
||||
put the derived-row writer back in competition with the legacy pipeline.
|
||||
"""
|
||||
if not await knowledge_store_enabled_for(workspace_id):
|
||||
logger.info("Workspace %s is not git-backed; not indexing", workspace_id)
|
||||
return 0
|
||||
session_maker = get_celery_session_maker()
|
||||
async with session_maker() as session:
|
||||
try:
|
||||
outcome = await (
|
||||
index_tree(session, workspace_id)
|
||||
if full
|
||||
else index_changes(session, workspace_id)
|
||||
)
|
||||
except KnowledgeStoreLockError:
|
||||
if full:
|
||||
# A competing rebuild converges the same tree; racing it would
|
||||
# just serialize two identical rebuilds. The sweep re-drives
|
||||
# this workspace if anything was missed.
|
||||
logger.info(
|
||||
"Workspace %s is already being indexed; skipping this rebuild",
|
||||
workspace_id,
|
||||
)
|
||||
return 0
|
||||
# Incremental: propagate so the task retries after the holder is
|
||||
# done — a save landing mid-converge must not wait for the sweep.
|
||||
raise
|
||||
return outcome.indexed
|
||||
|
||||
|
||||
async def _sweep() -> int:
|
||||
"""Enqueue indexing wherever the stamp disagrees with the store's HEAD.
|
||||
|
||||
Candidates are **flipped** workspaces only (`workspaces.knowledge_store_enabled`).
|
||||
A seeded-but-unflipped workspace has a repo on disk too, but Postgres is
|
||||
still its write model — indexing it would fight the legacy pipeline.
|
||||
"""
|
||||
session_maker = get_celery_session_maker()
|
||||
async with session_maker() as session:
|
||||
result = await session.execute(
|
||||
select(Workspace.id, Workspace.last_indexed_revision).where(
|
||||
Workspace.knowledge_store_enabled.is_(True)
|
||||
)
|
||||
)
|
||||
stamps = dict(result.all())
|
||||
|
||||
enqueued = 0
|
||||
for workspace_id, stamp in stamps.items():
|
||||
if enqueued >= SWEEP_ENQUEUE_CAP:
|
||||
logger.info(
|
||||
"Drift sweep hit its cap of %d; the rest wait for the next run",
|
||||
SWEEP_ENQUEUE_CAP,
|
||||
)
|
||||
break
|
||||
head = await KnowledgeStore.for_workspace(workspace_id).get_current_revision()
|
||||
if head is None or head == stamp:
|
||||
continue
|
||||
if stamp is None:
|
||||
# Never indexed: a full converge that embeds the whole tree. Route
|
||||
# it with the rebuilds so a backfill can't bury user-facing saves.
|
||||
reindex_knowledge_store.delay(workspace_id)
|
||||
else:
|
||||
index_knowledge_store_revision.delay(workspace_id)
|
||||
enqueued += 1
|
||||
|
||||
if enqueued:
|
||||
logger.info("Drift sweep enqueued indexing for %d workspaces", enqueued)
|
||||
return enqueued
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
"""Celery task pruning abandoned knowledge-store working copies."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from app.celery_app import celery_app
|
||||
from app.knowledge_store.janitor import prune_abandoned_working_copies
|
||||
from app.knowledge_store.settings import load_knowledge_store_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@celery_app.task(name="prune_knowledge_store_working_copies")
|
||||
def prune_knowledge_store_working_copies() -> int:
|
||||
if not load_knowledge_store_settings().enabled:
|
||||
return 0
|
||||
pruned = asyncio.run(prune_abandoned_working_copies())
|
||||
total = sum(len(ids) for ids in pruned.values())
|
||||
if pruned:
|
||||
logger.info("Pruned %d abandoned working copies: %s", total, pruned)
|
||||
return total
|
||||
|
|
@ -14,11 +14,7 @@ from typing import Any
|
|||
from app.agents.chat.multi_agent_chat.main_agent.middleware.kb_persistence import (
|
||||
commit_staged_filesystem_state,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.main_agent.middleware.knowledge_store_persistence import (
|
||||
commit_turn_working_copy,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.filesystem_selection import FilesystemMode
|
||||
from app.knowledge_store.settings import knowledge_store_enabled_for
|
||||
from app.services.new_streaming_service import VercelStreamingService
|
||||
from app.tasks.chat.message_parts_normalizer import (
|
||||
final_assistant_parts_from_messages,
|
||||
|
|
@ -123,46 +119,6 @@ async def stream_agent_events(
|
|||
except Exception as exc:
|
||||
_perf_log.warning("[stream_agent_events] safety-net commit failed: %s", exc)
|
||||
|
||||
# A turn paused for approval is not a finished turn: the graph resumes into
|
||||
# this same working copy, so the copy has to outlive the stream.
|
||||
pending_values = all_interrupt_values(state)
|
||||
|
||||
# Same safety net for the git-native path. The pending state is the turn's
|
||||
# working copy on disk, so no state markers gate it: no copy (or aafter_agent
|
||||
# already committed and discarded it) means the call is a no-op. No LLM on
|
||||
# this path — the commit gets the deterministic fallback message.
|
||||
#
|
||||
# Skipped while paused, because the helper both commits and discards: it
|
||||
# would cut the turn's writes so far into a revision of their own, and drop
|
||||
# a folder the agent made on the way — unrecoverable, git storing no empty
|
||||
# directories, which then fails the write the approval was granted for. The
|
||||
# legacy net above keeps running: under this backend the tools stage nothing,
|
||||
# so it is already a no-op here, and gating it would change the path that is
|
||||
# still live for workspaces not yet flipped.
|
||||
if (
|
||||
not pending_values
|
||||
and fallback_commit_filesystem_mode == FilesystemMode.CLOUD
|
||||
and fallback_commit_workspace_id is not None
|
||||
and await knowledge_store_enabled_for(fallback_commit_workspace_id)
|
||||
):
|
||||
try:
|
||||
delta = await commit_turn_working_copy(
|
||||
workspace_id=fallback_commit_workspace_id,
|
||||
thread_id=fallback_commit_thread_id,
|
||||
created_by_id=fallback_commit_created_by_id,
|
||||
llm=None,
|
||||
)
|
||||
if delta:
|
||||
await agent.aupdate_state(
|
||||
config,
|
||||
delta,
|
||||
as_node="KnowledgeStorePersistenceMiddleware.after_agent",
|
||||
)
|
||||
except Exception as exc:
|
||||
_perf_log.warning(
|
||||
"[stream_agent_events] git-native safety-net commit failed: %s", exc
|
||||
)
|
||||
|
||||
contract_state = state_values.get("file_operation_contract") or {}
|
||||
contract_turn_id = contract_state.get("turn_id")
|
||||
current_turn_id = config.get("configurable", {}).get("turn_id", "")
|
||||
|
|
@ -214,6 +170,7 @@ async def stream_agent_events(
|
|||
result.accumulated_text = accumulated_text
|
||||
log_file_contract("turn_outcome", result)
|
||||
|
||||
pending_values = all_interrupt_values(state)
|
||||
if pending_values:
|
||||
result.is_interrupted = True
|
||||
# One frame per paused subagent so each parallel HITL renders its own
|
||||
|
|
|
|||
|
|
@ -87,7 +87,6 @@ dependencies = [
|
|||
"croniter>=2.0.0",
|
||||
"scrapling[fetchers]>=0.4.11",
|
||||
"posthog>=6.0.0",
|
||||
"dulwich>=0.22.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
|
|
@ -235,7 +234,7 @@ python_functions = ["test_*"]
|
|||
addopts = "-v --tb=short -x --strict-markers -ra --durations=5 --import-mode=importlib"
|
||||
markers = [
|
||||
"unit: pure logic tests, no DB or external services",
|
||||
"integration: tests that require real infrastructure (PostgreSQL, Redis)"
|
||||
"integration: tests that require a real PostgreSQL database"
|
||||
]
|
||||
filterwarnings = [
|
||||
"ignore::UserWarning:chonkie",
|
||||
|
|
|
|||
|
|
@ -1,141 +0,0 @@
|
|||
"""Fleet runner for the Phase 5 knowledge-store migration.
|
||||
|
||||
Dry run by default: reports parity per workspace, writes nothing. Re-run with
|
||||
--yes to seed for real, and --yes --flip to also turn seeded workspaces
|
||||
git-native (only ever on a passing parity report). Every report is appended
|
||||
to a JSONL file, so a fleet pass is resumable and auditable; re-seeding is
|
||||
idempotent and convergent, so re-running after a partial pass only heals.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
from dataclasses import asdict
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import select, update
|
||||
|
||||
from app.db import Workspace, async_session_maker
|
||||
from app.knowledge_store.migrate import migrate_workspace
|
||||
from app.knowledge_store.store import KnowledgeStore
|
||||
|
||||
|
||||
async def _workspace_ids(only: list[int]) -> list[int]:
|
||||
if only:
|
||||
return only
|
||||
async with async_session_maker() as session:
|
||||
rows = await session.execute(select(Workspace.id).order_by(Workspace.id))
|
||||
return [row[0] for row in rows]
|
||||
|
||||
|
||||
async def _set_flip(workspace_id: int, enabled: bool) -> None:
|
||||
"""Flip one workspace, carrying its index stamp with it.
|
||||
|
||||
Stamping the store's head on the way in is what keeps the seed's promise:
|
||||
passing parity *is* the assertion that the chunk index already matches this
|
||||
revision, so a NULL stamp would have the drift sweep read the workspace as
|
||||
never-indexed and re-embed the whole tree — the cost the seed exists to
|
||||
avoid. Read from head rather than the report's ``seeded_revision``, which is
|
||||
``None`` on an idempotent re-seed.
|
||||
|
||||
Clearing it on the way out forces a full converge if the workspace is ever
|
||||
flipped back, since the legacy pipeline owned the chunks in between.
|
||||
"""
|
||||
revision = (
|
||||
await KnowledgeStore.for_workspace(workspace_id).get_current_revision()
|
||||
if enabled
|
||||
else None
|
||||
)
|
||||
async with async_session_maker() as session:
|
||||
await session.execute(
|
||||
update(Workspace)
|
||||
.where(Workspace.id == workspace_id)
|
||||
.values(knowledge_store_enabled=enabled, last_indexed_revision=revision)
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--yes",
|
||||
action="store_true",
|
||||
help="Actually seed. Without this flag the command is a parity dry run.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--workspace",
|
||||
type=int,
|
||||
action="append",
|
||||
default=[],
|
||||
help="Limit to this workspace id (repeatable). Default: all workspaces.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--out",
|
||||
default="knowledge_store_migration_reports.jsonl",
|
||||
help="JSONL file the per-workspace reports are appended to.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--flip",
|
||||
action="store_true",
|
||||
help="With --yes: turn each workspace git-native after its parity passes.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--unflip",
|
||||
action="store_true",
|
||||
help="Roll the listed workspaces back to the old write path. Does nothing else.",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.unflip:
|
||||
if not args.workspace:
|
||||
raise SystemExit("--unflip requires explicit --workspace ids")
|
||||
for workspace_id in args.workspace:
|
||||
await _set_flip(workspace_id, False)
|
||||
print(f"workspace {workspace_id}: rolled back to the old write path")
|
||||
return
|
||||
if args.flip and not args.yes:
|
||||
raise SystemExit("--flip requires --yes (never flip on a dry run)")
|
||||
|
||||
ids = await _workspace_ids(args.workspace)
|
||||
ok = failed = 0
|
||||
with open(args.out, "a") as out:
|
||||
for workspace_id in ids:
|
||||
# Fresh session per workspace: one failed workspace must not
|
||||
# poison the session the rest of the fleet reads through.
|
||||
async with async_session_maker() as session:
|
||||
report = await migrate_workspace(
|
||||
session, workspace_id, dry_run=not args.yes
|
||||
)
|
||||
out.write(
|
||||
json.dumps({"at": datetime.now(UTC).isoformat(), **asdict(report)})
|
||||
+ "\n"
|
||||
)
|
||||
out.flush()
|
||||
ok += report.ok
|
||||
failed += not report.ok
|
||||
if report.ok:
|
||||
status = "ok"
|
||||
if args.flip:
|
||||
await _set_flip(workspace_id, True)
|
||||
status = "ok, flipped git-native"
|
||||
elif report.error:
|
||||
status = f"error: {report.error}"
|
||||
else:
|
||||
# Expected on a pre-seed dry run: everything reads as missing.
|
||||
status = (
|
||||
f"drift: missing={len(report.missing)}"
|
||||
f" extra={len(report.extra)}"
|
||||
f" mismatched={len(report.mismatched)}"
|
||||
)
|
||||
print(f"workspace {workspace_id}: {status}, {report.files} file(s)")
|
||||
|
||||
mode = "seeded" if args.yes else "dry run"
|
||||
print(f"{mode}: {ok} ok, {failed} failed of {len(ids)}; reports in {args.out}")
|
||||
if failed:
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
|
|
@ -136,28 +136,6 @@ def _derivation_caches_disabled(monkeypatch):
|
|||
monkeypatch.setattr(app_config, "EMBEDDING_CACHE_ENABLED", False)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def workspace_flip(monkeypatch):
|
||||
"""Control the per-workspace knowledge-store flag for every workspace.
|
||||
|
||||
``_read_workspace_flag`` opens its own session, which cannot see rows
|
||||
inside the test transaction — so the DB read is the seam, exactly as in
|
||||
``tests/unit/knowledge_store/test_settings.py``.
|
||||
"""
|
||||
import app.knowledge_store.settings as ks_settings
|
||||
|
||||
ks_settings._flag_cache.clear()
|
||||
|
||||
def _set(enabled: bool) -> None:
|
||||
async def read(workspace_id: int) -> bool:
|
||||
return enabled
|
||||
|
||||
monkeypatch.setattr(ks_settings, "_read_workspace_flag", read)
|
||||
|
||||
yield _set
|
||||
ks_settings._flag_cache.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def patched_embed_texts(monkeypatch) -> MagicMock:
|
||||
mock = MagicMock(side_effect=lambda texts: [[0.1] * _EMBEDDING_DIM for _ in texts])
|
||||
|
|
|
|||
|
|
@ -1,24 +0,0 @@
|
|||
"""Integration conftest — prerequisites: Redis (``REDIS_APP_URL``) only.
|
||||
|
||||
Contenders normally wait 10s before giving up; tests shrink that window so
|
||||
contention cases resolve quickly. The give-up behavior itself is unchanged.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
import app.knowledge_store.write_lock as write_lock
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def workspace_id() -> str:
|
||||
"""Unique per test so runs never contend with each other or stale keys."""
|
||||
return f"it-{uuid.uuid4().hex}"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def short_lock_wait(monkeypatch):
|
||||
monkeypatch.setattr(write_lock, "LOCK_WAIT_SECONDS", 0.2)
|
||||
|
|
@ -1,496 +0,0 @@
|
|||
"""Postgres converges to the store: real git engine, real Redis lock, real DB.
|
||||
|
||||
The indexer's whole job is a projection, so these tests assert on the projection
|
||||
(rows, ids, chunk ids, the drift stamp) rather than on how it got there.
|
||||
|
||||
Paths carry ``.xml`` because that is what every real writer produces —
|
||||
``safe_filename`` appends it — and the extension is what ``parse_documents_path``
|
||||
strips to recover a title.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from app.config import config as app_config
|
||||
from app.db import Chunk, Document, DocumentStatus, DocumentType, DocumentVersion
|
||||
from app.indexing_pipeline.connector_document import ConnectorDocument
|
||||
from app.indexing_pipeline.indexing_pipeline_service import IndexingPipelineService
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.knowledge_store.identities import AGENT_IDENTITY, user_identity
|
||||
from app.knowledge_store.index.converge import PATH_MARKER, index_changes, index_tree
|
||||
from app.utils.document_converters import generate_unique_identifier_hash
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
# Content for the move tests. Git recognises a moved file by its content, so what
|
||||
# matters is that the same bytes land at the new path.
|
||||
MOVABLE = "# Content\n\na body git can match at its new path\n"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def knowledge_root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store(knowledge_root, db_workspace):
|
||||
return KnowledgeStore.for_workspace(db_workspace.id)
|
||||
|
||||
|
||||
async def commit(store, writes=None, removes=(), author=None):
|
||||
"""Record one revision; ``writes`` maps store path to markdown."""
|
||||
async with store.transaction(
|
||||
message="test", author=author or user_identity("1")
|
||||
) as tx:
|
||||
for path, markdown in (writes or {}).items():
|
||||
tx.write(path, markdown.encode())
|
||||
for path in removes:
|
||||
tx.remove(path)
|
||||
return tx.revision
|
||||
|
||||
|
||||
async def titles(session, workspace_id) -> dict[str, Document]:
|
||||
"""Every document in the workspace, keyed by title."""
|
||||
result = await session.execute(
|
||||
select(Document).where(Document.workspace_id == workspace_id)
|
||||
)
|
||||
return {document.title: document for document in result.scalars()}
|
||||
|
||||
|
||||
async def chunk_ids(session, document_id) -> list[int]:
|
||||
result = await session.execute(
|
||||
select(Chunk.id).where(Chunk.document_id == document_id).order_by(Chunk.id)
|
||||
)
|
||||
return list(result.scalars())
|
||||
|
||||
|
||||
async def versions(session, document_id) -> list[int]:
|
||||
result = await session.execute(
|
||||
select(DocumentVersion.version_number)
|
||||
.where(DocumentVersion.document_id == document_id)
|
||||
.order_by(DocumentVersion.version_number)
|
||||
)
|
||||
return list(result.scalars())
|
||||
|
||||
|
||||
async def chunk_shapes(session, document_id) -> list[tuple]:
|
||||
"""Text and line span of every chunk, in document order."""
|
||||
result = await session.execute(
|
||||
select(Chunk.content, Chunk.start_line, Chunk.end_line)
|
||||
.where(Chunk.document_id == document_id)
|
||||
.order_by(Chunk.position)
|
||||
)
|
||||
return list(result.all())
|
||||
|
||||
|
||||
# ── Identity ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_identical_content_at_two_paths_yields_two_documents(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
"""The case prepare_for_indexing collapses; `cp a b` is legal git."""
|
||||
await commit(store, {"documents/a.xml": "# Same", "documents/b.xml": "# Same"})
|
||||
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
rows = await titles(db_session, db_workspace.id)
|
||||
assert set(rows) == {"a", "b"}
|
||||
assert rows["a"].id != rows["b"].id
|
||||
|
||||
|
||||
async def test_uploaded_file_is_adopted_not_duplicated(
|
||||
store, db_session, db_workspace, db_user, patched_embed_texts
|
||||
):
|
||||
"""An upload already has a row under its own identity; indexing must reuse it."""
|
||||
upload = Document(
|
||||
title="report.pdf",
|
||||
document_type=DocumentType.FILE,
|
||||
document_metadata={"FILE_NAME": "report.pdf"},
|
||||
content="# Report",
|
||||
content_hash=f"hash-{uuid.uuid4().hex}",
|
||||
unique_identifier_hash=generate_unique_identifier_hash(
|
||||
DocumentType.FILE, "report.pdf", db_workspace.id
|
||||
),
|
||||
source_markdown="# Report",
|
||||
workspace_id=db_workspace.id,
|
||||
created_by_id=db_user.id,
|
||||
status=DocumentStatus.ready(),
|
||||
)
|
||||
db_session.add(upload)
|
||||
await db_session.flush()
|
||||
upload_id, upload_hash = upload.id, upload.unique_identifier_hash
|
||||
|
||||
await commit(store, {"documents/report.pdf.xml": "# Report"})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
total = await db_session.scalar(
|
||||
select(func.count(Document.id)).where(Document.workspace_id == db_workspace.id)
|
||||
)
|
||||
assert total == 1
|
||||
await db_session.refresh(upload)
|
||||
assert upload.id == upload_id
|
||||
# Identity and type stay the upload's; only the location marker is added.
|
||||
assert upload.document_type == DocumentType.FILE
|
||||
assert upload.unique_identifier_hash == upload_hash
|
||||
assert upload.document_metadata[PATH_MARKER] == "/documents/report.pdf.xml"
|
||||
assert upload.document_metadata["FILE_NAME"] == "report.pdf"
|
||||
|
||||
|
||||
async def test_existing_path_updates_in_place(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
"""A second index of the same path is an update, not a unique-hash collision."""
|
||||
await commit(store, {"documents/note.xml": "# First"})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
first_id = (await titles(db_session, db_workspace.id))["note"].id
|
||||
|
||||
await commit(store, {"documents/note.xml": "# First\n\nSecond paragraph."})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
rows = await titles(db_session, db_workspace.id)
|
||||
assert len(rows) == 1
|
||||
assert rows["note"].id == first_id
|
||||
assert "Second paragraph." in rows["note"].source_markdown
|
||||
|
||||
|
||||
async def test_a_document_in_a_folder_lands_under_that_folder(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
await commit(store, {"documents/Research/paper.xml": "# Paper"})
|
||||
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
row = (await titles(db_session, db_workspace.id))["paper"]
|
||||
assert row.folder_id is not None
|
||||
assert row.document_metadata[PATH_MARKER] == "/documents/Research/paper.xml"
|
||||
|
||||
|
||||
# ── Chunk reuse and removal ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_editing_one_file_leaves_another_files_chunks_untouched(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
await commit(
|
||||
store, {"documents/a.xml": "# A\n\nAlpha.", "documents/b.xml": "# B\n\nBeta."}
|
||||
)
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
untouched = (await titles(db_session, db_workspace.id))["b"]
|
||||
before = await chunk_ids(db_session, untouched.id)
|
||||
assert before
|
||||
|
||||
await commit(store, {"documents/a.xml": "# A\n\nAlpha, revised."})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
assert await chunk_ids(db_session, untouched.id) == before
|
||||
|
||||
|
||||
async def test_removed_path_deletes_the_document_and_its_chunks(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
await commit(store, {"documents/a.xml": "# A", "documents/b.xml": "# B"})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
doomed_id = (await titles(db_session, db_workspace.id))["a"].id
|
||||
assert await chunk_ids(db_session, doomed_id)
|
||||
|
||||
await commit(store, removes=["documents/a.xml"])
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
assert set(await titles(db_session, db_workspace.id)) == {"b"}
|
||||
assert await chunk_ids(db_session, doomed_id) == []
|
||||
|
||||
|
||||
async def test_a_move_keeps_the_document_and_its_history(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
"""The row has to outlive a move, not just be replaced by an equivalent one.
|
||||
Its version history cascades from the id, so a new row means an agent moving
|
||||
a file silently destroys every saved version of it — and dangles the
|
||||
``document_id`` in citations already written into past answers."""
|
||||
await commit(store, {"documents/old.xml": MOVABLE})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
document_id = (await titles(db_session, db_workspace.id))["old"].id
|
||||
db_session.add(
|
||||
DocumentVersion(
|
||||
document_id=document_id,
|
||||
version_number=1,
|
||||
source_markdown=MOVABLE,
|
||||
content_hash=f"hash-{uuid.uuid4().hex}",
|
||||
title="old",
|
||||
)
|
||||
)
|
||||
await db_session.commit()
|
||||
|
||||
await commit(store, {"documents/new.xml": MOVABLE}, removes=["documents/old.xml"])
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
rows = await titles(db_session, db_workspace.id)
|
||||
assert set(rows) == {"new"}
|
||||
assert rows["new"].id == document_id
|
||||
assert await versions(db_session, document_id) == [1]
|
||||
|
||||
|
||||
async def test_a_new_file_at_a_moved_from_path_gets_its_own_row(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
"""The moved row's fallback identity has to travel with it. Left behind on the
|
||||
old path, it makes the next document written there resolve to the moved row —
|
||||
one row claiming two paths, and the new file never getting a row of its own."""
|
||||
await commit(store, {"documents/old.xml": MOVABLE})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
moved_id = (await titles(db_session, db_workspace.id))["old"].id
|
||||
|
||||
await commit(store, {"documents/new.xml": MOVABLE}, removes=["documents/old.xml"])
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
await commit(store, {"documents/old.xml": "# Reused path\n\nnew note here\n"})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
rows = await titles(db_session, db_workspace.id)
|
||||
assert set(rows) == {"new", "old"}
|
||||
assert rows["new"].id == moved_id
|
||||
assert rows["old"].id != moved_id
|
||||
|
||||
|
||||
async def test_a_move_that_rewrites_the_file_keeps_the_marked_document(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
"""With nothing in common git sees no move, so the halves arrive separately —
|
||||
and an editor retitle has already moved the marker while leaving
|
||||
unique_identifier_hash behind. The removal then resolves by that hash to the
|
||||
row the upsert just updated, dropping a document whose file is still in the
|
||||
tree, invisible until the next full rebuild."""
|
||||
await commit(store, {"documents/old.xml": MOVABLE})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
row = (await titles(db_session, db_workspace.id))["old"]
|
||||
document_id = row.id
|
||||
row.document_metadata = {**row.document_metadata, PATH_MARKER: "/documents/new.xml"}
|
||||
await db_session.commit()
|
||||
|
||||
await commit(
|
||||
store,
|
||||
{"documents/new.xml": "# Other\n\nnothing whatever in common\n"},
|
||||
removes=["documents/old.xml"],
|
||||
)
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
rows = await titles(db_session, db_workspace.id)
|
||||
assert set(rows) == {"new"}
|
||||
assert rows["new"].id == document_id
|
||||
|
||||
|
||||
# ── Convergence ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_reindex_keeps_document_ids(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
"""Rebuild replaces chunks, never document rows: their ids reach the browser."""
|
||||
await commit(store, {"documents/a.xml": "# A", "documents/notes/b.xml": "# B"})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
before = {t: d.id for t, d in (await titles(db_session, db_workspace.id)).items()}
|
||||
|
||||
await index_tree(db_session, db_workspace.id)
|
||||
|
||||
after = {t: d.id for t, d in (await titles(db_session, db_workspace.id)).items()}
|
||||
assert after == before
|
||||
|
||||
|
||||
async def test_reindex_reaches_the_same_state_as_the_incremental_path(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
await commit(store, {"documents/a.xml": "# A", "documents/b.xml": "# B"})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
await commit(
|
||||
store, {"documents/b.xml": "# B\n\nEdited."}, removes=["documents/a.xml"]
|
||||
)
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
incremental = {
|
||||
title: document.source_markdown
|
||||
for title, document in (await titles(db_session, db_workspace.id)).items()
|
||||
}
|
||||
|
||||
await index_tree(db_session, db_workspace.id)
|
||||
|
||||
rebuilt = {
|
||||
title: document.source_markdown
|
||||
for title, document in (await titles(db_session, db_workspace.id)).items()
|
||||
}
|
||||
assert rebuilt == incremental
|
||||
|
||||
|
||||
async def test_indexing_a_stamped_revision_does_nothing(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
await commit(store, {"documents/a.xml": "# A"})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
calls = patched_embed_texts.call_count
|
||||
|
||||
outcome = await index_changes(db_session, db_workspace.id)
|
||||
|
||||
assert outcome.indexed == 0
|
||||
assert patched_embed_texts.call_count == calls
|
||||
|
||||
|
||||
async def test_a_missed_revision_is_folded_in_by_the_next_run(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
"""A dropped task must not strand its revision: the next run converges both."""
|
||||
await commit(store, {"documents/a.xml": "# A"})
|
||||
await commit(store, {"documents/b.xml": "# B"}) # no index run for this one
|
||||
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
assert set(await titles(db_session, db_workspace.id)) == {"a", "b"}
|
||||
await db_session.refresh(db_workspace)
|
||||
assert db_workspace.last_indexed_revision == await store.get_current_revision()
|
||||
|
||||
|
||||
async def test_a_connector_document_survives_a_rebuild(
|
||||
store, db_session, db_workspace, db_user, patched_embed_texts
|
||||
):
|
||||
"""Prune is keyed on the ownership marker; connector rows have no path at all."""
|
||||
connector_row = Document(
|
||||
title="Slack thread",
|
||||
document_type=DocumentType.SLACK_CONNECTOR,
|
||||
document_metadata={"channel": "general"},
|
||||
content="Hello",
|
||||
content_hash=f"hash-{uuid.uuid4().hex}",
|
||||
unique_identifier_hash=f"unique-{uuid.uuid4().hex}",
|
||||
source_markdown="Hello",
|
||||
workspace_id=db_workspace.id,
|
||||
created_by_id=db_user.id,
|
||||
status=DocumentStatus.ready(),
|
||||
)
|
||||
db_session.add(connector_row)
|
||||
await db_session.flush()
|
||||
|
||||
await commit(store, {"documents/a.xml": "# A"})
|
||||
await index_tree(db_session, db_workspace.id)
|
||||
|
||||
assert "Slack thread" in await titles(db_session, db_workspace.id)
|
||||
|
||||
|
||||
async def test_a_rebuild_prunes_a_row_whose_file_is_gone(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
"""The rebuild path has no change list, so the tree is the only authority."""
|
||||
await commit(store, {"documents/a.xml": "# A", "documents/b.xml": "# B"})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
# Remove the file without letting the incremental path see the removal.
|
||||
await commit(store, removes=["documents/a.xml"])
|
||||
await index_tree(db_session, db_workspace.id)
|
||||
|
||||
assert set(await titles(db_session, db_workspace.id)) == {"b"}
|
||||
|
||||
|
||||
# ── Authorship, skips, failures ─────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_an_agent_authored_revision_falls_back_to_the_workspace_owner(
|
||||
store, db_session, db_workspace, db_user, patched_embed_texts
|
||||
):
|
||||
"""Autonomous writes carry no user id, and created_by_id rejects blanks."""
|
||||
await commit(
|
||||
store, {"documents/agent.xml": "# Written by the agent"}, author=AGENT_IDENTITY
|
||||
)
|
||||
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
|
||||
assert (await titles(db_session, db_workspace.id))["agent"].created_by_id == (
|
||||
db_user.id
|
||||
)
|
||||
|
||||
|
||||
async def test_unusable_blobs_are_skipped_and_the_stamp_still_advances(
|
||||
store, db_session, db_workspace, patched_embed_texts
|
||||
):
|
||||
"""A touched file and a binary are legal git; neither may wedge the workspace."""
|
||||
async with store.transaction(message="mixed", author=user_identity("1")) as tx:
|
||||
tx.write("documents/good.xml", b"# Good")
|
||||
tx.write("documents/blank.xml", b" \n\n")
|
||||
tx.write("documents/binary.xml", b"\xff\xfe\x00\x01")
|
||||
revision = tx.revision
|
||||
|
||||
outcome = await index_changes(db_session, db_workspace.id)
|
||||
|
||||
assert set(await titles(db_session, db_workspace.id)) == {"good"}
|
||||
assert (outcome.indexed, outcome.skipped) == (1, 2)
|
||||
await db_session.refresh(db_workspace)
|
||||
assert db_workspace.last_indexed_revision == revision
|
||||
|
||||
|
||||
async def test_the_git_path_chunks_identically_to_the_connector_path(
|
||||
store, db_session, db_workspace, db_user, patched_embed_texts
|
||||
):
|
||||
"""Differential, not a golden baseline: a baseline rots on the first re-chunk.
|
||||
|
||||
This is also the only assertion that the shared chunk/cache/reconcile chain —
|
||||
which ships to every workspace, flag or no flag — still produces the same
|
||||
chunks and the same line spans it did for connectors.
|
||||
"""
|
||||
markdown = (
|
||||
"# Report\n"
|
||||
"\n"
|
||||
"First paragraph with some detail.\n"
|
||||
"\n"
|
||||
"| col | val |\n"
|
||||
"| --- | --- |\n"
|
||||
"| a | 1 |\n"
|
||||
"\n"
|
||||
"Closing paragraph.\n"
|
||||
)
|
||||
through_connector = Document(
|
||||
title="Report via connector",
|
||||
document_type=DocumentType.SLACK_CONNECTOR,
|
||||
document_metadata={},
|
||||
content=markdown,
|
||||
content_hash=f"hash-{uuid.uuid4().hex}",
|
||||
unique_identifier_hash=f"unique-{uuid.uuid4().hex}",
|
||||
source_markdown=markdown,
|
||||
workspace_id=db_workspace.id,
|
||||
created_by_id=db_user.id,
|
||||
status=DocumentStatus.pending(),
|
||||
)
|
||||
db_session.add(through_connector)
|
||||
await db_session.flush()
|
||||
await IndexingPipelineService(db_session).index(
|
||||
through_connector,
|
||||
ConnectorDocument(
|
||||
title=through_connector.title,
|
||||
source_markdown=markdown,
|
||||
unique_id="slack-1",
|
||||
document_type=DocumentType.SLACK_CONNECTOR,
|
||||
workspace_id=db_workspace.id,
|
||||
created_by_id=str(db_user.id),
|
||||
),
|
||||
)
|
||||
|
||||
await commit(store, {"documents/Report.xml": markdown})
|
||||
await index_changes(db_session, db_workspace.id)
|
||||
through_git = (await titles(db_session, db_workspace.id))["Report"]
|
||||
|
||||
assert await chunk_shapes(db_session, through_git.id) == await chunk_shapes(
|
||||
db_session, through_connector.id
|
||||
)
|
||||
|
||||
|
||||
async def test_a_failed_document_withholds_the_stamp(
|
||||
store, db_session, db_workspace, patched_embed_texts_raises
|
||||
):
|
||||
"""Without this the sweep has nothing to retry and the gap is permanent."""
|
||||
await commit(store, {"documents/a.xml": "# A"})
|
||||
|
||||
outcome = await index_changes(db_session, db_workspace.id)
|
||||
|
||||
assert outcome.failed == 1
|
||||
assert outcome.stamped is False
|
||||
await db_session.refresh(db_workspace)
|
||||
assert db_workspace.last_indexed_revision is None
|
||||
|
|
@ -1,264 +0,0 @@
|
|||
"""The always-on parity check: real documents, real repos, real comparison.
|
||||
|
||||
This is what replaces reading migration reports by hand once workspaces are
|
||||
flipped, so its verdict has to be trustworthy on its own — a check that says
|
||||
``ok`` while git and Postgres disagree is worse than no check at all.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
|
||||
import app.tasks.celery_tasks.knowledge_store.drift_monitor_task as monitor
|
||||
from app.config import config as app_config
|
||||
from app.db import Document, DocumentStatus, DocumentType, Workspace
|
||||
from app.knowledge_store.migrate import migrate_workspace
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def knowledge_root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def session_on_test_connection(db_session, monkeypatch):
|
||||
"""The monitor opens its own sessions; point them at the test transaction.
|
||||
|
||||
Patched on ``app.db`` rather than the task module because the monitor
|
||||
imports the maker at call time, per workspace.
|
||||
"""
|
||||
import app.db as db
|
||||
|
||||
maker = async_sessionmaker(
|
||||
bind=db_session.bind,
|
||||
expire_on_commit=False,
|
||||
join_transaction_mode="create_savepoint",
|
||||
)
|
||||
monkeypatch.setattr(db, "async_session_maker", maker)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def repairs_enqueued(monkeypatch):
|
||||
"""Record repairs instead of handing them to a broker that is not running.
|
||||
|
||||
Autouse because ``.delay`` is the one outbound call in this module: a test
|
||||
that drifts without the seam in place would reach for a real broker.
|
||||
"""
|
||||
calls: list[int] = []
|
||||
|
||||
monkeypatch.setattr(
|
||||
monitor.reindex_knowledge_store,
|
||||
"delay",
|
||||
lambda workspace_id: calls.append(workspace_id),
|
||||
)
|
||||
return calls
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def drift_metrics(monkeypatch):
|
||||
"""Record what the monitor reports, in place of an OTel exporter."""
|
||||
recorded: list[tuple[int, str]] = []
|
||||
|
||||
monkeypatch.setattr(
|
||||
monitor.metrics,
|
||||
"record_knowledge_store_drift_check",
|
||||
lambda *, workspace_id, status: recorded.append((workspace_id, status)),
|
||||
)
|
||||
return recorded
|
||||
|
||||
|
||||
async def make_workspace(session, user_id, *, flipped: bool) -> Workspace:
|
||||
space = Workspace(name="Watched", user_id=user_id, knowledge_store_enabled=flipped)
|
||||
session.add(space)
|
||||
await session.flush()
|
||||
return space
|
||||
|
||||
|
||||
async def add_document(session, workspace, user_id) -> Document:
|
||||
document = Document(
|
||||
title="Note",
|
||||
document_type=DocumentType.NOTE,
|
||||
document_metadata={},
|
||||
content="# Note",
|
||||
content_hash=f"hash-{uuid.uuid4().hex}",
|
||||
unique_identifier_hash=f"unique-{uuid.uuid4().hex}",
|
||||
source_markdown="# Note\n\nBody.",
|
||||
workspace_id=workspace.id,
|
||||
created_by_id=user_id,
|
||||
status=DocumentStatus.ready(),
|
||||
)
|
||||
session.add(document)
|
||||
await session.flush()
|
||||
return document
|
||||
|
||||
|
||||
async def test_a_seeded_workspace_reports_ok(
|
||||
db_session, db_user, knowledge_root, session_on_test_connection, drift_metrics
|
||||
):
|
||||
space = await make_workspace(db_session, db_user.id, flipped=True)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
await migrate_workspace(db_session, space.id)
|
||||
|
||||
assert await monitor._check_flipped_workspaces() == {"ok": 1}
|
||||
assert drift_metrics == [(space.id, "ok")]
|
||||
|
||||
|
||||
async def test_a_document_missing_from_the_store_reports_drift(
|
||||
db_session, db_user, knowledge_root, session_on_test_connection, drift_metrics
|
||||
):
|
||||
"""Postgres has content git never received — the exact case to alarm on."""
|
||||
space = await make_workspace(db_session, db_user.id, flipped=True)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
await migrate_workspace(db_session, space.id)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
|
||||
assert await monitor._check_flipped_workspaces() == {"drift": 1}
|
||||
assert drift_metrics == [(space.id, "drift")]
|
||||
|
||||
|
||||
async def test_an_unflipped_workspace_is_not_checked(
|
||||
db_session, db_user, knowledge_root, session_on_test_connection, drift_metrics
|
||||
):
|
||||
"""Postgres is still its write model, so disagreeing with git is expected."""
|
||||
space = await make_workspace(db_session, db_user.id, flipped=False)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
|
||||
assert await monitor._check_flipped_workspaces() == {}
|
||||
assert drift_metrics == []
|
||||
|
||||
|
||||
async def test_a_failed_check_is_reported_and_the_sweep_continues(
|
||||
db_session,
|
||||
db_user,
|
||||
knowledge_root,
|
||||
session_on_test_connection,
|
||||
drift_metrics,
|
||||
monkeypatch,
|
||||
):
|
||||
"""A workspace the check cannot read is its own status, not a lost run.
|
||||
|
||||
The fresh session per workspace exists so one failure cannot poison the
|
||||
rest; a silent stop here would leave every later workspace unchecked while
|
||||
the task still looks healthy.
|
||||
"""
|
||||
broken = await make_workspace(db_session, db_user.id, flipped=True)
|
||||
healthy = await make_workspace(db_session, db_user.id, flipped=True)
|
||||
await add_document(db_session, healthy, db_user.id)
|
||||
await migrate_workspace(db_session, healthy.id)
|
||||
|
||||
real = monitor.migrate_workspace
|
||||
|
||||
async def fail_on_broken(session, workspace_id, **kwargs):
|
||||
report = await real(session, workspace_id, **kwargs)
|
||||
if workspace_id == broken.id:
|
||||
return replace(report, error="repo unreadable")
|
||||
return report
|
||||
|
||||
monkeypatch.setattr(monitor, "migrate_workspace", fail_on_broken)
|
||||
|
||||
assert await monitor._check_flipped_workspaces() == {"error": 1, "ok": 1}
|
||||
assert (healthy.id, "ok") in drift_metrics
|
||||
|
||||
|
||||
# ── Repair ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_drift_enqueues_a_whole_tree_converge(
|
||||
db_session,
|
||||
db_user,
|
||||
knowledge_root,
|
||||
session_on_test_connection,
|
||||
drift_metrics,
|
||||
repairs_enqueued,
|
||||
):
|
||||
"""The hourly sweep cannot see this drift, so the alarm has to close it.
|
||||
|
||||
Both sides of the sweep's comparison are git revisions, which leaves
|
||||
Postgres-side disagreement to this check alone; if it only alarmed, repair
|
||||
would depend on someone reading the alert.
|
||||
"""
|
||||
space = await make_workspace(db_session, db_user.id, flipped=True)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
await migrate_workspace(db_session, space.id)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
|
||||
assert await monitor._check_flipped_workspaces() == {"drift": 1}
|
||||
assert repairs_enqueued == [space.id]
|
||||
|
||||
|
||||
async def test_a_workspace_in_parity_is_not_repaired(
|
||||
db_session,
|
||||
db_user,
|
||||
knowledge_root,
|
||||
session_on_test_connection,
|
||||
drift_metrics,
|
||||
repairs_enqueued,
|
||||
):
|
||||
"""A daily whole-tree converge per healthy workspace is the cost of a bug here."""
|
||||
space = await make_workspace(db_session, db_user.id, flipped=True)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
await migrate_workspace(db_session, space.id)
|
||||
|
||||
assert await monitor._check_flipped_workspaces() == {"ok": 1}
|
||||
assert repairs_enqueued == []
|
||||
|
||||
|
||||
async def test_a_failed_check_alarms_without_repairing(
|
||||
db_session,
|
||||
db_user,
|
||||
knowledge_root,
|
||||
session_on_test_connection,
|
||||
drift_metrics,
|
||||
repairs_enqueued,
|
||||
monkeypatch,
|
||||
):
|
||||
"""A store the check could not read will not be fixed by indexing it harder.
|
||||
|
||||
Repairing on ``error`` would also mean repairing on a verdict nobody
|
||||
computed: the parity fields of a failed report describe nothing.
|
||||
"""
|
||||
space = await make_workspace(db_session, db_user.id, flipped=True)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
|
||||
real = monitor.migrate_workspace
|
||||
|
||||
async def fail(session, workspace_id, **kwargs):
|
||||
return replace(await real(session, workspace_id, **kwargs), error="unreadable")
|
||||
|
||||
monkeypatch.setattr(monitor, "migrate_workspace", fail)
|
||||
|
||||
assert await monitor._check_flipped_workspaces() == {"error": 1}
|
||||
assert repairs_enqueued == []
|
||||
|
||||
|
||||
async def test_the_cap_bounds_repairs_not_checks(
|
||||
db_session,
|
||||
db_user,
|
||||
knowledge_root,
|
||||
session_on_test_connection,
|
||||
drift_metrics,
|
||||
repairs_enqueued,
|
||||
monkeypatch,
|
||||
):
|
||||
"""Fleet-wide drift is a systemic fault; fanning out rebuilds compounds it.
|
||||
|
||||
Every workspace is still checked and still alarms — only the repair is
|
||||
capped, so the signal stays complete while the work stays bounded.
|
||||
"""
|
||||
monkeypatch.setattr(monitor, "REPAIR_ENQUEUE_CAP", 1)
|
||||
for _ in range(2):
|
||||
space = await make_workspace(db_session, db_user.id, flipped=True)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
await migrate_workspace(db_session, space.id)
|
||||
await add_document(db_session, space, db_user.id)
|
||||
|
||||
assert await monitor._check_flipped_workspaces() == {"drift": 2}
|
||||
assert len(repairs_enqueued) == 1
|
||||
|
|
@ -1,162 +0,0 @@
|
|||
"""The hourly sweep: the only recovery for indexing lost to a broker or worker.
|
||||
|
||||
Every other trigger is fire-and-forget, so if the sweep picks the wrong
|
||||
candidates or routes them to the wrong queue, a workspace can sit stale
|
||||
indefinitely with nothing to notice. These tests use real git repos and real
|
||||
rows; the seam is ``.delay``, the outbound boundary where a task leaves for a
|
||||
broker that is not running here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
import app.tasks.celery_tasks.knowledge_store.index_tasks as index_tasks
|
||||
from app.config import config as app_config
|
||||
from app.db import Workspace
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.knowledge_store.identities import user_identity
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def knowledge_root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def celery_session_on_test_connection(db_session, monkeypatch):
|
||||
"""Point the sweep's own session maker at the test transaction."""
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
|
||||
maker = async_sessionmaker(
|
||||
bind=db_session.bind,
|
||||
expire_on_commit=False,
|
||||
join_transaction_mode="create_savepoint",
|
||||
)
|
||||
monkeypatch.setattr(index_tasks, "get_celery_session_maker", lambda: maker)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def enqueued(monkeypatch):
|
||||
"""Record which task each workspace was handed to, in call order."""
|
||||
calls: list[tuple[str, int]] = []
|
||||
|
||||
def spy(name):
|
||||
return lambda workspace_id: calls.append((name, workspace_id))
|
||||
|
||||
monkeypatch.setattr(
|
||||
index_tasks.index_knowledge_store_revision, "delay", spy("incremental")
|
||||
)
|
||||
monkeypatch.setattr(index_tasks.reindex_knowledge_store, "delay", spy("rebuild"))
|
||||
return calls
|
||||
|
||||
|
||||
async def make_workspace(session, user_id, *, flipped: bool, stamp: str | None = None):
|
||||
space = Workspace(
|
||||
name="Swept",
|
||||
user_id=user_id,
|
||||
knowledge_store_enabled=flipped,
|
||||
last_indexed_revision=stamp,
|
||||
)
|
||||
session.add(space)
|
||||
await session.flush()
|
||||
return space
|
||||
|
||||
|
||||
async def commit(workspace_id) -> str:
|
||||
"""Give a workspace a store with one revision in it."""
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
async with store.transaction(message="seed", author=user_identity("1")) as tx:
|
||||
tx.write("documents/a.xml", b"# A")
|
||||
return tx.revision
|
||||
|
||||
|
||||
# ── Candidate selection ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_a_flipped_workspace_trailing_its_store_is_enqueued(
|
||||
db_session, db_user, knowledge_root, celery_session_on_test_connection, enqueued
|
||||
):
|
||||
space = await make_workspace(db_session, db_user.id, flipped=True, stamp="stale")
|
||||
await commit(space.id)
|
||||
|
||||
assert await index_tasks._sweep() == 1
|
||||
assert enqueued == [("incremental", space.id)]
|
||||
|
||||
|
||||
async def test_a_workspace_level_with_its_store_is_left_alone(
|
||||
db_session, db_user, knowledge_root, celery_session_on_test_connection, enqueued
|
||||
):
|
||||
space = await make_workspace(db_session, db_user.id, flipped=True)
|
||||
space.last_indexed_revision = await commit(space.id)
|
||||
await db_session.flush()
|
||||
|
||||
assert await index_tasks._sweep() == 0
|
||||
assert enqueued == []
|
||||
|
||||
|
||||
async def test_an_unflipped_workspace_is_never_a_candidate(
|
||||
db_session, db_user, knowledge_root, celery_session_on_test_connection, enqueued
|
||||
):
|
||||
"""It may have a seeded repo, but Postgres is still its write model."""
|
||||
space = await make_workspace(db_session, db_user.id, flipped=False, stamp="stale")
|
||||
await commit(space.id)
|
||||
|
||||
assert await index_tasks._sweep() == 0
|
||||
assert enqueued == []
|
||||
|
||||
|
||||
async def test_a_workspace_with_no_store_yet_is_skipped(
|
||||
db_session, db_user, knowledge_root, celery_session_on_test_connection, enqueued
|
||||
):
|
||||
"""Flipped before its seed lands: nothing to converge to, so no task."""
|
||||
await make_workspace(db_session, db_user.id, flipped=True)
|
||||
|
||||
assert await index_tasks._sweep() == 0
|
||||
assert enqueued == []
|
||||
|
||||
|
||||
# ── Routing ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_a_never_indexed_workspace_routes_to_the_rebuild_task(
|
||||
db_session, db_user, knowledge_root, celery_session_on_test_connection, enqueued
|
||||
):
|
||||
"""A NULL stamp means embedding the whole tree.
|
||||
|
||||
That belongs on the connectors queue with the other rebuilds; sending it to
|
||||
the per-save task would put a fleet-wide backfill ahead of user-facing saves
|
||||
on the fast queue.
|
||||
"""
|
||||
space = await make_workspace(db_session, db_user.id, flipped=True, stamp=None)
|
||||
await commit(space.id)
|
||||
|
||||
assert await index_tasks._sweep() == 1
|
||||
assert enqueued == [("rebuild", space.id)]
|
||||
|
||||
|
||||
# ── Fan-out ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_the_cap_bounds_the_fan_out_not_the_check(
|
||||
db_session,
|
||||
db_user,
|
||||
knowledge_root,
|
||||
celery_session_on_test_connection,
|
||||
enqueued,
|
||||
monkeypatch,
|
||||
):
|
||||
"""Two drifted workspaces, room for one: the other waits for the next run."""
|
||||
monkeypatch.setattr(index_tasks, "SWEEP_ENQUEUE_CAP", 1)
|
||||
for _ in range(2):
|
||||
space = await make_workspace(
|
||||
db_session, db_user.id, flipped=True, stamp="stale"
|
||||
)
|
||||
await commit(space.id)
|
||||
|
||||
assert await index_tasks._sweep() == 1
|
||||
assert len(enqueued) == 1
|
||||
|
|
@ -1,325 +0,0 @@
|
|||
"""End-of-turn commit body: working-copy diff → one revision → receipts.
|
||||
|
||||
Real git engine + real Redis write lock; only the LLM boundary is faked.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from langchain_core.language_models.fake_chat_models import FakeListChatModel
|
||||
|
||||
from app.agents.chat.multi_agent_chat.main_agent.middleware.knowledge_store_persistence.commit_turn import (
|
||||
commit_turn_working_copy,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.filesystem.backends.git_tree import (
|
||||
GitTreeBackend,
|
||||
)
|
||||
from app.config import config as app_config
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.knowledge_store.write_lock import workspace_write_lock
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
THREAD_ID = 42
|
||||
USER_ID = "1"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def knowledge_root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def llm() -> FakeListChatModel:
|
||||
return FakeListChatModel(responses=["docs: capture turn output"])
|
||||
|
||||
|
||||
async def _turn_writes(workspace_id, files: dict[str, bytes]) -> KnowledgeStore:
|
||||
"""Simulate a turn: materialize the thread's working copy and write into it."""
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
copy = await store.open_working_copy(f"thread-{THREAD_ID}")
|
||||
for rel, content in files.items():
|
||||
target = copy.path / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes(content)
|
||||
return store
|
||||
|
||||
|
||||
async def _commit(workspace_id, llm):
|
||||
return await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
|
||||
|
||||
async def _committed_turn(workspace_id, llm, files: dict[str, bytes]) -> KnowledgeStore:
|
||||
"""Leave the store one revision in, so the next turn can edit or drop files."""
|
||||
store = await _turn_writes(workspace_id, files)
|
||||
await _commit(workspace_id, llm)
|
||||
return store
|
||||
|
||||
|
||||
async def _next_turn_copy(store: KnowledgeStore):
|
||||
"""A fresh copy for the following turn, materialized from the committed tree."""
|
||||
return await store.open_working_copy(f"thread-{THREAD_ID}")
|
||||
|
||||
|
||||
def _operations(delta) -> dict[str, str]:
|
||||
return {r["preview"]: r["operation"] for r in delta["receipts"]}
|
||||
|
||||
|
||||
class _Runtime:
|
||||
"""Enough of ``ToolRuntime`` for the backend to resolve a working copy."""
|
||||
|
||||
def __init__(self, thread_id: str) -> None:
|
||||
self.config = {"configurable": {"thread_id": thread_id}}
|
||||
self.state: dict = {}
|
||||
self.tool_call_id = "call_x"
|
||||
|
||||
|
||||
async def test_commits_the_turns_net_changes_as_one_revision(
|
||||
knowledge_root, workspace_id, llm
|
||||
):
|
||||
store = await _turn_writes(workspace_id, {"documents/note.md": b"hello"})
|
||||
|
||||
delta = await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
|
||||
revisions = await store.list_revisions()
|
||||
assert len(revisions) == 1
|
||||
assert await store.read_as_of(revisions[0].id, "documents/note.md") == b"hello"
|
||||
|
||||
receipts = delta["receipts"]
|
||||
assert [r["status"] for r in receipts] == ["success"]
|
||||
assert receipts[0]["external_id"] == revisions[0].id
|
||||
assert receipts[0]["operation"] == "write_file"
|
||||
|
||||
|
||||
async def test_message_carries_subject_and_thread_trailer(
|
||||
knowledge_root, workspace_id, llm
|
||||
):
|
||||
store = await _turn_writes(workspace_id, {"documents/note.md": b"hello"})
|
||||
|
||||
await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
|
||||
message = (await store.list_revisions())[0].message
|
||||
assert message.startswith("docs: capture turn output")
|
||||
assert f"Thread: {THREAD_ID}" in message
|
||||
|
||||
|
||||
async def test_attributes_author_to_user_and_committer_to_agent(
|
||||
knowledge_root, workspace_id, llm
|
||||
):
|
||||
store = await _turn_writes(workspace_id, {"documents/note.md": b"hello"})
|
||||
|
||||
await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
|
||||
rev = (await store.list_revisions())[0]
|
||||
assert USER_ID in rev.author
|
||||
assert rev.author != rev.committer
|
||||
assert "agent" in rev.committer.lower()
|
||||
|
||||
|
||||
async def test_discards_the_copy_so_a_second_commit_is_a_no_op(
|
||||
knowledge_root, workspace_id, llm
|
||||
):
|
||||
await _turn_writes(workspace_id, {"documents/note.md": b"hello"})
|
||||
|
||||
first = await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
second = await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
|
||||
assert first is not None
|
||||
assert second is None
|
||||
|
||||
|
||||
async def test_a_turn_that_never_touched_the_store_commits_nothing(
|
||||
knowledge_root, workspace_id, llm
|
||||
):
|
||||
delta = await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
|
||||
assert delta is None
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
assert await store.get_current_revision() is None
|
||||
|
||||
|
||||
async def test_an_untouched_copy_records_nothing(knowledge_root, workspace_id, llm):
|
||||
store = await _turn_writes(workspace_id, {})
|
||||
|
||||
delta = await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
|
||||
assert delta is None
|
||||
assert await store.get_current_revision() is None
|
||||
|
||||
|
||||
async def test_lock_contention_yields_failed_receipts_and_keeps_the_copy(
|
||||
knowledge_root, workspace_id, llm, short_lock_wait
|
||||
):
|
||||
store = await _turn_writes(workspace_id, {"documents/note.md": b"hello"})
|
||||
|
||||
async with workspace_write_lock(workspace_id):
|
||||
delta = await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
|
||||
assert [r["status"] for r in delta["receipts"]] == ["failed"]
|
||||
assert await store.get_current_revision() is None
|
||||
# The copy survives, so the thread's next turn commits the leftover work.
|
||||
retry = await commit_turn_working_copy(
|
||||
workspace_id=workspace_id,
|
||||
thread_id=THREAD_ID,
|
||||
created_by_id=USER_ID,
|
||||
llm=llm,
|
||||
)
|
||||
assert [r["status"] for r in retry["receipts"]] == ["success"]
|
||||
|
||||
|
||||
# --- Every kind of change a turn can make ---
|
||||
#
|
||||
# Receipts are the orchestrator's ground truth for what the agent did, so the
|
||||
# operation each kind maps to has to be pinned per kind — a turn that only ever
|
||||
# adds files leaves the modification and removal mappings unverified.
|
||||
|
||||
|
||||
async def test_an_edited_file_is_recorded_as_a_modification(
|
||||
knowledge_root, workspace_id, llm
|
||||
):
|
||||
store = await _committed_turn(workspace_id, llm, {"documents/note.md": b"hello"})
|
||||
|
||||
copy = await _next_turn_copy(store)
|
||||
(copy.path / "documents/note.md").write_bytes(b"hello again")
|
||||
delta = await _commit(workspace_id, llm)
|
||||
|
||||
assert _operations(delta) == {"documents/note.md": "edit_file"}
|
||||
revision = (await store.list_revisions())[0].id
|
||||
assert await store.read_as_of(revision, "documents/note.md") == b"hello again"
|
||||
|
||||
|
||||
async def test_a_deleted_file_is_recorded_as_a_removal(
|
||||
knowledge_root, workspace_id, llm
|
||||
):
|
||||
store = await _committed_turn(workspace_id, llm, {"documents/note.md": b"hello"})
|
||||
|
||||
copy = await _next_turn_copy(store)
|
||||
(copy.path / "documents/note.md").unlink()
|
||||
delta = await _commit(workspace_id, llm)
|
||||
|
||||
assert _operations(delta) == {"documents/note.md": "rm"}
|
||||
revision = (await store.list_revisions())[0].id
|
||||
assert await store.list_paths(revision) == []
|
||||
|
||||
|
||||
async def test_a_moved_file_is_recorded_as_one_move(knowledge_root, workspace_id, llm):
|
||||
"""A move is committed as a removal plus a write, but read back as a rename:
|
||||
git recognises the content, which is what lets the index move the document's
|
||||
row instead of replacing it. The receipt reports the move it was."""
|
||||
store = await _committed_turn(workspace_id, llm, {"documents/old.md": b"hello"})
|
||||
|
||||
copy = await _next_turn_copy(store)
|
||||
(copy.path / "documents/old.md").rename(copy.path / "documents/new.md")
|
||||
delta = await _commit(workspace_id, llm)
|
||||
|
||||
assert _operations(delta) == {"documents/new.md": "move_file"}
|
||||
revision = (await store.list_revisions())[0].id
|
||||
assert [e.path for e in await store.list_paths(revision)] == ["documents/new.md"]
|
||||
|
||||
|
||||
async def test_a_mixed_turn_records_one_receipt_per_change(
|
||||
knowledge_root, workspace_id, llm
|
||||
):
|
||||
store = await _committed_turn(
|
||||
workspace_id, llm, {"documents/kept.md": b"a", "documents/dropped.md": b"b"}
|
||||
)
|
||||
|
||||
copy = await _next_turn_copy(store)
|
||||
(copy.path / "documents/kept.md").write_bytes(b"a edited")
|
||||
(copy.path / "documents/dropped.md").unlink()
|
||||
(copy.path / "documents/added.md").write_bytes(b"c")
|
||||
delta = await _commit(workspace_id, llm)
|
||||
|
||||
assert _operations(delta) == {
|
||||
"documents/kept.md": "edit_file",
|
||||
"documents/dropped.md": "rm",
|
||||
"documents/added.md": "write_file",
|
||||
}
|
||||
assert len(await store.list_revisions()) == 2
|
||||
|
||||
|
||||
async def test_a_delegated_write_is_committed_by_the_parent_turn(
|
||||
knowledge_root, workspace_id, llm
|
||||
):
|
||||
"""A subagent's write must reach the turn's revision, not vanish.
|
||||
|
||||
Writes through the backend rather than a hand-built copy id: the defect was
|
||||
the backend and the commit deriving that id differently.
|
||||
"""
|
||||
backend = GitTreeBackend(workspace_id, _Runtime(f"{THREAD_ID}::task:call_x"))
|
||||
await backend.awrite("/documents/delegated.md", "from the subagent")
|
||||
|
||||
delta = await _commit(workspace_id, llm)
|
||||
|
||||
assert _operations(delta) == {"documents/delegated.md": "write_file"}
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
revision = (await store.list_revisions())[0].id
|
||||
assert (
|
||||
await store.read_as_of(revision, "documents/delegated.md")
|
||||
== b"from the subagent"
|
||||
)
|
||||
# The copy is discarded, not left behind as an orphan.
|
||||
assert not (
|
||||
knowledge_root / ".working_copies" / str(workspace_id) / f"thread-{THREAD_ID}"
|
||||
).exists()
|
||||
|
||||
|
||||
async def test_failed_receipts_cover_removals_too(
|
||||
knowledge_root, workspace_id, llm, short_lock_wait
|
||||
):
|
||||
store = await _committed_turn(workspace_id, llm, {"documents/note.md": b"hello"})
|
||||
|
||||
copy = await _next_turn_copy(store)
|
||||
(copy.path / "documents/note.md").unlink()
|
||||
async with workspace_write_lock(workspace_id):
|
||||
delta = await _commit(workspace_id, llm)
|
||||
|
||||
assert _operations(delta) == {"documents/note.md": "rm"}
|
||||
assert [r["status"] for r in delta["receipts"]] == ["failed"]
|
||||
|
|
@ -1,439 +0,0 @@
|
|||
"""Document saves become store revisions (real git engine + real Redis lock)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
from app.agents.chat.runtime.path_resolver import PATH_MARKER
|
||||
from app.config import config as app_config
|
||||
from app.db import Document, DocumentStatus, DocumentType
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.services import document_revision_recorder as recorder
|
||||
from app.services.document_revision_recorder import (
|
||||
record_markdown_files,
|
||||
record_prepared_documents,
|
||||
record_saved_document,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def knowledge_root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
async def test_one_save_records_one_revision(knowledge_root, workspace_id):
|
||||
revision = await record_markdown_files(
|
||||
workspace_id=workspace_id,
|
||||
files={"documents/notes/meeting.md": "# Meeting"},
|
||||
message="docs: save meeting.md",
|
||||
author_user_id="1",
|
||||
)
|
||||
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
assert revision is not None
|
||||
assert (
|
||||
await store.read_as_of(revision, "documents/notes/meeting.md") == b"# Meeting"
|
||||
)
|
||||
rev = (await store.list_revisions())[0]
|
||||
assert "1" in rev.author
|
||||
assert "meeting.md" in rev.message
|
||||
|
||||
|
||||
async def test_a_sync_batch_records_one_revision(knowledge_root, workspace_id):
|
||||
revision = await record_markdown_files(
|
||||
workspace_id=workspace_id,
|
||||
files={
|
||||
"documents/notion/roadmap.md": "# Roadmap",
|
||||
"documents/notion/okrs.md": "# OKRs",
|
||||
},
|
||||
message="sync: index 2 document(s)",
|
||||
author_user_id="1",
|
||||
)
|
||||
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
revisions = await store.list_revisions()
|
||||
assert [r.id for r in revisions] == [revision]
|
||||
assert await store.read_as_of(revision, "documents/notion/okrs.md") == b"# OKRs"
|
||||
|
||||
|
||||
async def test_unchanged_content_records_nothing(knowledge_root, workspace_id):
|
||||
files = {"documents/notion/roadmap.md": "# Roadmap"}
|
||||
first = await record_markdown_files(
|
||||
workspace_id=workspace_id, files=files, message="sync", author_user_id="1"
|
||||
)
|
||||
second = await record_markdown_files(
|
||||
workspace_id=workspace_id, files=files, message="sync", author_user_id="1"
|
||||
)
|
||||
|
||||
assert first is not None
|
||||
assert second is None
|
||||
assert len(await KnowledgeStore.for_workspace(workspace_id).list_revisions()) == 1
|
||||
|
||||
|
||||
async def test_empty_batch_records_nothing(knowledge_root, workspace_id):
|
||||
revision = await record_markdown_files(
|
||||
workspace_id=workspace_id, files={}, message="sync", author_user_id="1"
|
||||
)
|
||||
|
||||
assert revision is None
|
||||
assert not (knowledge_root / str(workspace_id)).exists()
|
||||
|
||||
|
||||
async def test_a_retitled_document_leaves_no_file_at_its_old_path(
|
||||
knowledge_root, workspace_id
|
||||
):
|
||||
"""One document is one file. Without the removal a retitle forks it into two."""
|
||||
await record_markdown_files(
|
||||
workspace_id=workspace_id,
|
||||
files={"documents/Old title.xml": "# Body"},
|
||||
message="docs: save Old title.xml",
|
||||
author_user_id="1",
|
||||
)
|
||||
|
||||
revision = await record_markdown_files(
|
||||
workspace_id=workspace_id,
|
||||
files={"documents/New title.xml": "# Body"},
|
||||
message="docs: save New title.xml",
|
||||
author_user_id="1",
|
||||
removes=["documents/Old title.xml"],
|
||||
)
|
||||
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
paths = {entry.path for entry in await store.list_paths(revision)}
|
||||
assert paths == {"documents/New title.xml"}
|
||||
|
||||
|
||||
async def test_disabled_store_records_nothing(monkeypatch, tmp_path, workspace_id):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", False)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
|
||||
revision = await record_markdown_files(
|
||||
workspace_id=workspace_id,
|
||||
files={"documents/notes/meeting.md": "# Meeting"},
|
||||
message="docs: save meeting.md",
|
||||
author_user_id="1",
|
||||
)
|
||||
|
||||
assert revision is None
|
||||
assert not (tmp_path / str(workspace_id)).exists()
|
||||
|
||||
|
||||
# --- record_saved_document: the editor's path resolution and retitle handling ---
|
||||
#
|
||||
# The tests above hand `removes` in ready-made, so they only prove the batch
|
||||
# primitive honours it. These drive the caller that has to *derive* the removal
|
||||
# from the row's path marker — the step a retitle depends on.
|
||||
|
||||
|
||||
async def _make_document(session, workspace, user, title: str) -> Document:
|
||||
document = Document(
|
||||
title=title,
|
||||
document_type=DocumentType.NOTE,
|
||||
document_metadata={},
|
||||
content=f"# {title}",
|
||||
content_hash=f"hash-{uuid.uuid4().hex}",
|
||||
unique_identifier_hash=f"unique-{uuid.uuid4().hex}",
|
||||
source_markdown=f"# {title}",
|
||||
workspace_id=workspace.id,
|
||||
created_by_id=user.id,
|
||||
status=DocumentStatus.ready(),
|
||||
)
|
||||
session.add(document)
|
||||
await session.commit()
|
||||
return document
|
||||
|
||||
|
||||
async def _save(
|
||||
session,
|
||||
workspace,
|
||||
user,
|
||||
document,
|
||||
*,
|
||||
title,
|
||||
markdown="# Body",
|
||||
title_is_explicit=False,
|
||||
):
|
||||
return await record_saved_document(
|
||||
session,
|
||||
workspace_id=workspace.id,
|
||||
doc_id=document.id,
|
||||
title=title,
|
||||
folder_id=None,
|
||||
markdown=markdown,
|
||||
author_user_id=str(user.id),
|
||||
title_is_explicit=title_is_explicit,
|
||||
)
|
||||
|
||||
|
||||
async def _store_paths(workspace, revision: str) -> set[str]:
|
||||
store = KnowledgeStore.for_workspace(workspace.id)
|
||||
return {entry.path for entry in await store.list_paths(revision)}
|
||||
|
||||
|
||||
async def test_a_save_records_the_document_and_remembers_its_path(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
workspace_flip(True)
|
||||
document = await _make_document(db_session, db_workspace, db_user, "Meeting notes")
|
||||
|
||||
revision = await _save(
|
||||
db_session, db_workspace, db_user, document, title="Meeting notes"
|
||||
)
|
||||
|
||||
assert revision is not None
|
||||
paths = await _store_paths(db_workspace, revision)
|
||||
assert len(paths) == 1
|
||||
assert "Meeting notes" in next(iter(paths))
|
||||
# Remembered so the *next* save knows where the document used to live.
|
||||
assert "Meeting notes" in document.document_metadata[PATH_MARKER]
|
||||
|
||||
|
||||
async def test_a_retitle_leaves_only_the_new_path(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
"""The removal has to be derived from the marker, not supplied by the caller."""
|
||||
workspace_flip(True)
|
||||
document = await _make_document(db_session, db_workspace, db_user, "Old title")
|
||||
await _save(db_session, db_workspace, db_user, document, title="Old title")
|
||||
|
||||
document.title = "New title"
|
||||
revision = await _save(
|
||||
db_session,
|
||||
db_workspace,
|
||||
db_user,
|
||||
document,
|
||||
title="New title",
|
||||
title_is_explicit=True,
|
||||
)
|
||||
|
||||
paths = await _store_paths(db_workspace, revision)
|
||||
assert len(paths) == 1
|
||||
assert "New title" in next(iter(paths))
|
||||
assert "New title" in document.document_metadata[PATH_MARKER]
|
||||
|
||||
|
||||
async def test_a_retitle_records_the_move_as_one_revision(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
"""Two revisions, not three: the drop rides along with the write."""
|
||||
workspace_flip(True)
|
||||
document = await _make_document(db_session, db_workspace, db_user, "Old title")
|
||||
await _save(db_session, db_workspace, db_user, document, title="Old title")
|
||||
|
||||
document.title = "New title"
|
||||
await _save(
|
||||
db_session,
|
||||
db_workspace,
|
||||
db_user,
|
||||
document,
|
||||
title="New title",
|
||||
title_is_explicit=True,
|
||||
)
|
||||
|
||||
store = KnowledgeStore.for_workspace(db_workspace.id)
|
||||
assert len(await store.list_revisions()) == 2
|
||||
|
||||
|
||||
async def _agent_authored(session, workspace, user, path: str, markdown: str):
|
||||
"""A file under a name no title would derive, with the row that points at it."""
|
||||
await record_markdown_files(
|
||||
workspace_id=workspace.id,
|
||||
files={path.lstrip("/"): markdown},
|
||||
message="agent: write",
|
||||
author_user_id=str(user.id),
|
||||
)
|
||||
document = await _make_document(session, workspace, user, path.rsplit("/", 1)[-1])
|
||||
document.document_metadata = {PATH_MARKER: path}
|
||||
await session.commit()
|
||||
return document
|
||||
|
||||
|
||||
async def test_an_inferred_retitle_keeps_the_name_the_agent_chose(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
"""A note's title is re-read from its first heading on every save, so an
|
||||
ordinary save arrives here looking like a retitle. Placing by that title
|
||||
would rename the agent's file — and invalidate the path it is holding."""
|
||||
workspace_flip(True)
|
||||
document = await _agent_authored(
|
||||
db_session, db_workspace, db_user, "/documents/summary.md", "# Key Points"
|
||||
)
|
||||
|
||||
revision = await _save(
|
||||
db_session,
|
||||
db_workspace,
|
||||
db_user,
|
||||
document,
|
||||
title="Key Points",
|
||||
markdown="# Key Points\n\nEdited.",
|
||||
)
|
||||
|
||||
assert await _store_paths(db_workspace, revision) == {"documents/summary.md"}
|
||||
assert document.document_metadata[PATH_MARKER] == "/documents/summary.md"
|
||||
|
||||
|
||||
async def test_an_explicit_rename_still_moves_the_agent_s_file(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
"""The gate narrows which titles place a file; it does not stop a rename."""
|
||||
workspace_flip(True)
|
||||
document = await _agent_authored(
|
||||
db_session, db_workspace, db_user, "/documents/summary.md", "# Key Points"
|
||||
)
|
||||
|
||||
revision = await _save(
|
||||
db_session,
|
||||
db_workspace,
|
||||
db_user,
|
||||
document,
|
||||
title="Key Points",
|
||||
markdown="# Key Points\n\nEdited.",
|
||||
title_is_explicit=True,
|
||||
)
|
||||
|
||||
paths = await _store_paths(db_workspace, revision)
|
||||
assert paths == {"documents/Key Points.xml"}
|
||||
assert document.document_metadata[PATH_MARKER] == "/documents/Key Points.xml"
|
||||
|
||||
|
||||
async def test_no_marker_is_left_when_nothing_was_recorded(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
"""A marker without a file makes the row look indexer-owned, so a later
|
||||
whole-tree converge would prune it."""
|
||||
workspace_flip(True)
|
||||
document = await _make_document(db_session, db_workspace, db_user, "Meeting notes")
|
||||
await _save(db_session, db_workspace, db_user, document, title="Meeting notes")
|
||||
|
||||
document.document_metadata = {}
|
||||
await db_session.commit()
|
||||
revision = await _save(
|
||||
db_session, db_workspace, db_user, document, title="Meeting notes"
|
||||
)
|
||||
|
||||
assert revision is None
|
||||
assert PATH_MARKER not in document.document_metadata
|
||||
|
||||
|
||||
async def test_a_marker_outside_the_documents_namespace_is_not_dropped(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
"""Resolving it raises; swallowing that is what keeps the save recordable."""
|
||||
workspace_flip(True)
|
||||
document = await _make_document(db_session, db_workspace, db_user, "Meeting notes")
|
||||
document.document_metadata = {PATH_MARKER: "/elsewhere/foreign.xml"}
|
||||
await db_session.commit()
|
||||
|
||||
revision = await _save(
|
||||
db_session, db_workspace, db_user, document, title="Meeting notes"
|
||||
)
|
||||
|
||||
assert revision is not None
|
||||
assert len(await _store_paths(db_workspace, revision)) == 1
|
||||
|
||||
|
||||
async def test_a_save_in_an_unflipped_workspace_records_nothing(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
workspace_flip(False)
|
||||
document = await _make_document(db_session, db_workspace, db_user, "Meeting notes")
|
||||
|
||||
revision = await _save(
|
||||
db_session, db_workspace, db_user, document, title="Meeting notes"
|
||||
)
|
||||
|
||||
assert revision is None
|
||||
assert not (knowledge_root / str(db_workspace.id)).exists()
|
||||
|
||||
|
||||
async def test_a_recording_failure_does_not_fail_the_save(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip, monkeypatch
|
||||
):
|
||||
"""The Postgres save already committed; the store is the coexisting copy."""
|
||||
workspace_flip(True)
|
||||
document = await _make_document(db_session, db_workspace, db_user, "Meeting notes")
|
||||
|
||||
async def boom(**kwargs):
|
||||
raise RuntimeError("store unavailable")
|
||||
|
||||
monkeypatch.setattr(recorder, "record_markdown_files", boom)
|
||||
|
||||
revision = await _save(
|
||||
db_session, db_workspace, db_user, document, title="Meeting notes"
|
||||
)
|
||||
|
||||
assert revision is None
|
||||
assert PATH_MARKER not in document.document_metadata
|
||||
|
||||
|
||||
# --- record_prepared_documents: the connector sync batch ---
|
||||
|
||||
|
||||
async def test_a_sync_batch_records_every_document_with_markdown(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
workspace_flip(True)
|
||||
first = await _make_document(db_session, db_workspace, db_user, "Roadmap")
|
||||
second = await _make_document(db_session, db_workspace, db_user, "OKRs")
|
||||
|
||||
revision = await record_prepared_documents(db_session, [first, second])
|
||||
|
||||
assert revision is not None
|
||||
paths = await _store_paths(db_workspace, revision)
|
||||
assert len(paths) == 2
|
||||
assert any("Roadmap" in path for path in paths)
|
||||
|
||||
|
||||
async def test_a_document_without_markdown_is_left_out_of_the_batch(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
"""An extraction that produced nothing must not land as an empty file."""
|
||||
workspace_flip(True)
|
||||
kept = await _make_document(db_session, db_workspace, db_user, "Roadmap")
|
||||
empty = await _make_document(db_session, db_workspace, db_user, "Unextracted")
|
||||
empty.source_markdown = None
|
||||
await db_session.commit()
|
||||
|
||||
revision = await record_prepared_documents(db_session, [kept, empty])
|
||||
|
||||
paths = await _store_paths(db_workspace, revision)
|
||||
assert len(paths) == 1
|
||||
assert "Roadmap" in next(iter(paths))
|
||||
|
||||
|
||||
async def test_an_empty_sync_batch_records_nothing(
|
||||
knowledge_root, db_session, workspace_flip
|
||||
):
|
||||
workspace_flip(True)
|
||||
|
||||
assert await record_prepared_documents(db_session, []) is None
|
||||
|
||||
|
||||
async def test_a_sync_batch_in_an_unflipped_workspace_records_nothing(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip
|
||||
):
|
||||
workspace_flip(False)
|
||||
document = await _make_document(db_session, db_workspace, db_user, "Roadmap")
|
||||
|
||||
assert await record_prepared_documents(db_session, [document]) is None
|
||||
assert not (knowledge_root / str(db_workspace.id)).exists()
|
||||
|
||||
|
||||
async def test_a_sync_batch_failure_does_not_reach_the_caller(
|
||||
knowledge_root, db_session, db_workspace, db_user, workspace_flip, monkeypatch
|
||||
):
|
||||
workspace_flip(True)
|
||||
document = await _make_document(db_session, db_workspace, db_user, "Roadmap")
|
||||
|
||||
async def boom(**kwargs):
|
||||
raise RuntimeError("store unavailable")
|
||||
|
||||
monkeypatch.setattr(recorder, "record_markdown_files", boom)
|
||||
|
||||
assert await record_prepared_documents(db_session, [document]) is None
|
||||
|
|
@ -1,89 +0,0 @@
|
|||
"""Flipping a workspace also hands the derived index its starting point.
|
||||
|
||||
The seed deliberately never reindexes — it copies bytes out of Postgres, so the
|
||||
existing chunk index already matches what it wrote. A flip that left
|
||||
``last_indexed_revision`` NULL would throw that away: the drift sweep reads NULL
|
||||
as never-indexed and enqueues a whole-tree converge that re-embeds everything.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
|
||||
import scripts.migrate_knowledge_store as runner
|
||||
from app.config import config as app_config
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.knowledge_store.migrate import seed_workspace
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def knowledge_root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def runner_session_on_test_connection(db_session, monkeypatch):
|
||||
"""Point the runner's own session maker at the test transaction."""
|
||||
maker = async_sessionmaker(
|
||||
bind=db_session.bind,
|
||||
expire_on_commit=False,
|
||||
join_transaction_mode="create_savepoint",
|
||||
)
|
||||
monkeypatch.setattr(runner, "async_session_maker", maker)
|
||||
|
||||
|
||||
async def test_flipping_stamps_the_stores_head(
|
||||
knowledge_root, db_session, db_workspace, runner_session_on_test_connection
|
||||
):
|
||||
report = await seed_workspace(db_workspace.id, {"documents/a.xml": "# A"})
|
||||
|
||||
await runner._set_flip(db_workspace.id, True)
|
||||
|
||||
await db_session.refresh(db_workspace)
|
||||
assert db_workspace.knowledge_store_enabled is True
|
||||
assert db_workspace.last_indexed_revision == report.seeded_revision
|
||||
|
||||
|
||||
async def test_a_re_seed_that_recorded_nothing_still_stamps(
|
||||
knowledge_root, db_session, db_workspace, runner_session_on_test_connection
|
||||
):
|
||||
"""``seeded_revision`` is None on an idempotent re-seed; head is not."""
|
||||
await seed_workspace(db_workspace.id, {"documents/a.xml": "# A"})
|
||||
re_seed = await seed_workspace(db_workspace.id, {"documents/a.xml": "# A"})
|
||||
|
||||
await runner._set_flip(db_workspace.id, True)
|
||||
|
||||
await db_session.refresh(db_workspace)
|
||||
assert re_seed.seeded_revision is None
|
||||
head = await KnowledgeStore.for_workspace(db_workspace.id).get_current_revision()
|
||||
assert db_workspace.last_indexed_revision == head
|
||||
|
||||
|
||||
async def test_unflipping_clears_the_stamp(
|
||||
knowledge_root, db_session, db_workspace, runner_session_on_test_connection
|
||||
):
|
||||
"""Postgres owned the chunks while unflipped, so a re-flip must converge fully."""
|
||||
await seed_workspace(db_workspace.id, {"documents/a.xml": "# A"})
|
||||
await runner._set_flip(db_workspace.id, True)
|
||||
|
||||
await runner._set_flip(db_workspace.id, False)
|
||||
|
||||
await db_session.refresh(db_workspace)
|
||||
assert db_workspace.knowledge_store_enabled is False
|
||||
assert db_workspace.last_indexed_revision is None
|
||||
|
||||
|
||||
async def test_flipping_without_a_store_leaves_the_stamp_empty(
|
||||
knowledge_root, db_session, db_workspace, runner_session_on_test_connection
|
||||
):
|
||||
"""--flip only fires on passing parity, but reading a missing head must not raise."""
|
||||
await runner._set_flip(db_workspace.id, True)
|
||||
|
||||
await db_session.refresh(db_workspace)
|
||||
assert db_workspace.knowledge_store_enabled is True
|
||||
assert db_workspace.last_indexed_revision is None
|
||||
|
|
@ -1,155 +0,0 @@
|
|||
"""Phase 5 seeder: seed commit + byte parity (real git engine + real Redis lock)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import config as app_config
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.knowledge_store.identities import MIGRATION_IDENTITY
|
||||
from app.knowledge_store.migrate import seed_workspace
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
FILES = {
|
||||
"notes/roadmap.md": "# Roadmap",
|
||||
"notes/okrs.md": "# OKRs",
|
||||
"welcome.md": "# Welcome",
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def knowledge_root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
async def test_seed_records_one_revision_and_passes_parity(
|
||||
knowledge_root, workspace_id
|
||||
):
|
||||
report = await seed_workspace(workspace_id, FILES)
|
||||
|
||||
assert report.ok
|
||||
assert report.seeded_revision is not None
|
||||
assert report.files == 3
|
||||
assert report.missing == report.extra == report.mismatched == []
|
||||
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
revisions = await store.list_revisions()
|
||||
assert [r.id for r in revisions] == [report.seeded_revision]
|
||||
assert revisions[0].author == MIGRATION_IDENTITY
|
||||
assert await store.read_as_of(report.seeded_revision, "notes/okrs.md") == b"# OKRs"
|
||||
|
||||
|
||||
async def test_reseeding_unchanged_content_is_a_noop(knowledge_root, workspace_id):
|
||||
first = await seed_workspace(workspace_id, FILES)
|
||||
second = await seed_workspace(workspace_id, FILES)
|
||||
|
||||
assert second.ok
|
||||
assert second.seeded_revision is None
|
||||
revisions = await KnowledgeStore.for_workspace(workspace_id).list_revisions()
|
||||
assert [r.id for r in revisions] == [first.seeded_revision]
|
||||
|
||||
|
||||
async def test_dry_run_builds_nothing_and_reports_all_missing(
|
||||
knowledge_root, workspace_id
|
||||
):
|
||||
report = await seed_workspace(workspace_id, FILES, dry_run=True)
|
||||
|
||||
assert not report.ok
|
||||
assert report.seeded_revision is None
|
||||
assert sorted(report.missing) == sorted(FILES)
|
||||
assert not (knowledge_root / str(workspace_id)).exists()
|
||||
|
||||
|
||||
async def test_dry_run_against_a_seeded_store_passes_parity(
|
||||
knowledge_root, workspace_id
|
||||
):
|
||||
await seed_workspace(workspace_id, FILES)
|
||||
|
||||
report = await seed_workspace(workspace_id, FILES, dry_run=True)
|
||||
|
||||
assert report.ok
|
||||
assert report.seeded_revision is None
|
||||
|
||||
|
||||
async def test_reseeding_after_drift_converges(knowledge_root, workspace_id):
|
||||
"""Seed-then-flip-later: Postgres kept changing; a catch-up re-seed heals
|
||||
everything, including documents deleted since the first seed."""
|
||||
await seed_workspace(workspace_id, FILES)
|
||||
|
||||
drifted = dict(FILES)
|
||||
drifted.pop("welcome.md") # deleted in Postgres since the first seed
|
||||
drifted["notes/okrs.md"] = "# OKRs v2" # edited
|
||||
drifted["notes/new.md"] = "# New" # added
|
||||
report = await seed_workspace(workspace_id, drifted)
|
||||
|
||||
assert report.ok
|
||||
assert report.seeded_revision is not None
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
paths = {t.path for t in await store.list_paths(report.seeded_revision)}
|
||||
assert "welcome.md" not in paths
|
||||
assert (
|
||||
await store.read_as_of(report.seeded_revision, "notes/okrs.md") == b"# OKRs v2"
|
||||
)
|
||||
|
||||
|
||||
async def test_parity_names_missing_extra_and_mismatched_paths(
|
||||
knowledge_root, workspace_id
|
||||
):
|
||||
await seed_workspace(workspace_id, FILES)
|
||||
|
||||
drifted = dict(FILES)
|
||||
drifted.pop("welcome.md") # repo now has an extra path
|
||||
drifted["notes/okrs.md"] = "# OKRs v2" # repo content differs
|
||||
drifted["notes/new.md"] = "# New" # repo misses this path
|
||||
report = await seed_workspace(workspace_id, drifted, dry_run=True)
|
||||
|
||||
assert not report.ok
|
||||
assert report.missing == ["notes/new.md"]
|
||||
assert report.extra == ["welcome.md"]
|
||||
assert report.mismatched == ["notes/okrs.md"]
|
||||
|
||||
|
||||
async def test_expired_write_lock_lands_in_the_report(
|
||||
knowledge_root, workspace_id, monkeypatch
|
||||
):
|
||||
"""A real TTL expiry — the seed write outlives the Redis lock — is reported,
|
||||
never swallowed or raised past the fleet loop. The commit itself still
|
||||
landed (parity is clean), so ``error`` is the only trace of the lost hold."""
|
||||
import time
|
||||
|
||||
import app.knowledge_store.write_lock as write_lock
|
||||
from app.knowledge_store.engines.git import GitContentEngine
|
||||
|
||||
real_record = GitContentEngine.record
|
||||
|
||||
def slow_record(self, **kwargs):
|
||||
time.sleep(0.3) # outlive the shrunken TTL below
|
||||
return real_record(self, **kwargs)
|
||||
|
||||
monkeypatch.setattr(write_lock, "LOCK_TTL_SECONDS", 0.1)
|
||||
monkeypatch.setattr(GitContentEngine, "record", slow_record)
|
||||
report = await seed_workspace(workspace_id, FILES)
|
||||
|
||||
assert not report.ok
|
||||
assert "expired mid-block" in report.error
|
||||
assert report.seeded_revision is None
|
||||
assert report.missing == report.extra == report.mismatched == []
|
||||
|
||||
|
||||
async def test_any_seed_failure_is_contained_in_the_report(
|
||||
knowledge_root, workspace_id, monkeypatch
|
||||
):
|
||||
"""seed_workspace never raises: a crash anywhere (here the parity read)
|
||||
becomes ``error``, so one broken workspace can't abort a fleet run."""
|
||||
|
||||
def explode(self, revision):
|
||||
raise OSError("disk gone")
|
||||
|
||||
monkeypatch.setattr(KnowledgeStore, "list_paths", explode)
|
||||
report = await seed_workspace(workspace_id, FILES)
|
||||
|
||||
assert not report.ok
|
||||
assert report.error == "OSError: disk gone"
|
||||
assert report.files == len(FILES)
|
||||
|
|
@ -1,125 +0,0 @@
|
|||
"""Where the seeder puts a document, and what it records about it.
|
||||
|
||||
Placement is shared with every other writer, so the seeder must not re-invent it.
|
||||
Deriving a path from the title is only ever a guess: the agent's ``write_file``
|
||||
names its own files, so the store already holds names no title would produce.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from app.agents.chat.runtime.path_resolver import PATH_MARKER
|
||||
from app.config import config as app_config
|
||||
from app.db import Document, DocumentType
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.knowledge_store.migrate import migrate_workspace
|
||||
from app.utils.document_converters import generate_content_hash
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def knowledge_root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
async def _add_document(session, workspace, *, title, markdown, marker=None):
|
||||
document = Document(
|
||||
title=title,
|
||||
document_type=DocumentType.NOTE,
|
||||
document_metadata={PATH_MARKER: marker} if marker else {},
|
||||
content=markdown,
|
||||
content_hash=generate_content_hash(markdown, workspace.id),
|
||||
source_markdown=markdown,
|
||||
workspace_id=workspace.id,
|
||||
)
|
||||
session.add(document)
|
||||
await session.flush()
|
||||
return document
|
||||
|
||||
|
||||
async def test_an_agent_authored_name_is_seeded_where_it_already_lives(
|
||||
knowledge_root, db_session, db_workspace
|
||||
):
|
||||
"""The store holds ``canary.md``; the title would derive ``canary.md.xml``.
|
||||
Deriving would report false drift, and a real run would write the derived
|
||||
name and delete the agent's file as an orphan."""
|
||||
await _add_document(
|
||||
db_session,
|
||||
db_workspace,
|
||||
title="canary.md",
|
||||
markdown="# Canary",
|
||||
marker="/documents/canary.md",
|
||||
)
|
||||
|
||||
report = await migrate_workspace(db_session, db_workspace.id)
|
||||
|
||||
assert report.ok, report
|
||||
store = KnowledgeStore.for_workspace(db_workspace.id)
|
||||
paths = {t.path for t in await store.list_paths(report.seeded_revision)}
|
||||
assert paths == {"documents/canary.md"}
|
||||
|
||||
|
||||
async def test_a_row_with_no_marker_keeps_the_derived_name(
|
||||
knowledge_root, db_session, db_workspace
|
||||
):
|
||||
"""Every migrated row starts unmarked, so derivation stays the fallback and
|
||||
an existing store is not renamed out from under itself."""
|
||||
await _add_document(
|
||||
db_session, db_workspace, title="strategy.md", markdown="# Strategy"
|
||||
)
|
||||
|
||||
report = await migrate_workspace(db_session, db_workspace.id)
|
||||
|
||||
assert report.ok, report
|
||||
store = KnowledgeStore.for_workspace(db_workspace.id)
|
||||
paths = {t.path for t in await store.list_paths(report.seeded_revision)}
|
||||
assert paths == {"documents/strategy.md.xml"}
|
||||
|
||||
|
||||
async def test_seeding_records_the_path_it_wrote(
|
||||
knowledge_root, db_session, db_workspace
|
||||
):
|
||||
"""Without the marker a retitle cannot tell which file to drop from the
|
||||
tree, so it forks the document into two."""
|
||||
document = await _add_document(
|
||||
db_session, db_workspace, title="strategy.md", markdown="# Strategy"
|
||||
)
|
||||
assert PATH_MARKER not in (document.document_metadata or {})
|
||||
|
||||
await migrate_workspace(db_session, db_workspace.id)
|
||||
|
||||
await db_session.refresh(document)
|
||||
assert document.document_metadata[PATH_MARKER] == "/documents/strategy.md.xml"
|
||||
|
||||
|
||||
async def test_a_dry_run_records_nothing(knowledge_root, db_session, db_workspace):
|
||||
document = await _add_document(
|
||||
db_session, db_workspace, title="strategy.md", markdown="# Strategy"
|
||||
)
|
||||
|
||||
await migrate_workspace(db_session, db_workspace.id, dry_run=True)
|
||||
|
||||
await db_session.refresh(document)
|
||||
assert PATH_MARKER not in (document.document_metadata or {})
|
||||
|
||||
|
||||
async def test_the_marker_makes_a_re_seed_stable(
|
||||
knowledge_root, db_session, db_workspace
|
||||
):
|
||||
"""The path recorded by the first seed is what the second one reads back, so
|
||||
a workspace cannot drift to a new name just by being seeded twice."""
|
||||
await _add_document(
|
||||
db_session, db_workspace, title="strategy.md", markdown="# Strategy"
|
||||
)
|
||||
|
||||
first = await migrate_workspace(db_session, db_workspace.id)
|
||||
second = await migrate_workspace(db_session, db_workspace.id)
|
||||
|
||||
assert second.ok, second
|
||||
assert second.seeded_revision is None
|
||||
store = KnowledgeStore.for_workspace(db_workspace.id)
|
||||
paths = {t.path for t in await store.list_paths(first.seeded_revision)}
|
||||
assert paths == {"documents/strategy.md.xml"}
|
||||
|
|
@ -1,141 +0,0 @@
|
|||
"""What the end-of-stream safety net does to a turn that stopped for approval.
|
||||
|
||||
The net exists for the turn that dies mid-flight: it commits the working copy
|
||||
the ``aafter_agent`` hook never reached. A turn paused at an approval gate looks
|
||||
the same from here — the stream ends either way — but it is coming back, and the
|
||||
copy is the only place its work so far exists.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.filesystem_selection import FilesystemMode
|
||||
from app.config import config as app_config
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.services.new_streaming_service import VercelStreamingService
|
||||
from app.tasks.chat.streaming.agent.event_loop import stream_agent_events
|
||||
from app.tasks.chat.streaming.shared.stream_result import StreamResult
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
THREAD_ID = 4321
|
||||
COPY_ID = f"thread-{THREAD_ID}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Interrupt:
|
||||
value: dict[str, Any]
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Task:
|
||||
interrupts: tuple[_Interrupt, ...] = ()
|
||||
|
||||
|
||||
@dataclass
|
||||
class _State:
|
||||
"""Stand-in for the snapshot ``aget_state`` returns.
|
||||
|
||||
Empty ``values`` keeps the legacy staged-state net a no-op, so the only
|
||||
write path under test is the git-native one.
|
||||
"""
|
||||
|
||||
tasks: list[_Task] = field(default_factory=list)
|
||||
values: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class _Agent:
|
||||
"""Streams nothing; the turn's work is already on disk in the copy."""
|
||||
|
||||
def __init__(self, state: _State) -> None:
|
||||
self._state = state
|
||||
self.updates: list[Any] = []
|
||||
|
||||
async def astream_events(self, *_args, **_kwargs):
|
||||
return
|
||||
yield # pragma: no cover - makes this an async generator
|
||||
|
||||
async def aget_state(self, _config):
|
||||
return self._state
|
||||
|
||||
async def aupdate_state(self, _config, delta, **_kwargs):
|
||||
self.updates.append(delta)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def knowledge_root(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ROOT", str(tmp_path))
|
||||
return tmp_path
|
||||
|
||||
|
||||
async def _run_turn(workspace_id: int, state: _State) -> _Agent:
|
||||
agent = _Agent(state)
|
||||
async for _ in stream_agent_events(
|
||||
agent,
|
||||
{"configurable": {"thread_id": str(THREAD_ID)}},
|
||||
{},
|
||||
VercelStreamingService(),
|
||||
StreamResult(),
|
||||
fallback_commit_workspace_id=workspace_id,
|
||||
fallback_commit_created_by_id="1",
|
||||
fallback_commit_filesystem_mode=FilesystemMode.CLOUD,
|
||||
fallback_commit_thread_id=THREAD_ID,
|
||||
):
|
||||
pass
|
||||
return agent
|
||||
|
||||
|
||||
async def _copy_with_pending_work(workspace_id: int) -> tuple[Any, Any]:
|
||||
"""A copy holding one written file and one folder the agent just made."""
|
||||
store = KnowledgeStore.for_workspace(workspace_id)
|
||||
copy = await store.open_working_copy(COPY_ID)
|
||||
documents = copy.path / "documents"
|
||||
documents.mkdir(exist_ok=True)
|
||||
(documents / "draft.md").write_text("# Draft\n")
|
||||
(documents / "crud-test").mkdir(exist_ok=True)
|
||||
return store, copy
|
||||
|
||||
|
||||
async def test_a_turn_paused_for_approval_keeps_its_working_copy(
|
||||
knowledge_root, db_workspace, workspace_flip
|
||||
):
|
||||
"""The folder is the part that cannot be rebuilt: git stores no empty
|
||||
directory, so discarding here loses it for good — and the write the approval
|
||||
was granted for then fails for want of its parent."""
|
||||
workspace_flip(True)
|
||||
_, copy = await _copy_with_pending_work(db_workspace.id)
|
||||
pending_approval = {
|
||||
"type": "approval",
|
||||
"message": "Approve writing draft.md?",
|
||||
"action": {"name": "write_file", "args": {"file_path": "/documents/draft.md"}},
|
||||
"context": {},
|
||||
}
|
||||
paused = _State(tasks=[_Task(interrupts=(_Interrupt(value=pending_approval),))])
|
||||
|
||||
await _run_turn(db_workspace.id, paused)
|
||||
|
||||
assert copy.path.exists()
|
||||
assert (copy.path / "documents" / "draft.md").read_text() == "# Draft\n"
|
||||
assert (copy.path / "documents" / "crud-test").is_dir()
|
||||
|
||||
|
||||
async def test_a_finished_turn_still_gets_its_safety_net(
|
||||
knowledge_root, db_workspace, workspace_flip
|
||||
):
|
||||
"""The net's own reason for existing: no approval pending means the turn is
|
||||
over, and an uncommitted copy means the hook never ran."""
|
||||
workspace_flip(True)
|
||||
store, copy = await _copy_with_pending_work(db_workspace.id)
|
||||
|
||||
agent = await _run_turn(db_workspace.id, _State())
|
||||
|
||||
assert not copy.path.exists()
|
||||
revision = await store.get_current_revision()
|
||||
paths = {entry.path for entry in await store.list_paths(revision)}
|
||||
assert "documents/draft.md" in paths
|
||||
assert agent.updates, "the commit's delta should reach the graph"
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
"""KnowledgeStore facade end to end: real git engine + real Redis write lock."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from app.knowledge_store import KnowledgeStore
|
||||
from app.knowledge_store.engines.git import GitContentEngine
|
||||
from app.knowledge_store.write_lock import (
|
||||
KnowledgeStoreLockError,
|
||||
workspace_write_lock,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
AUTHOR = "SurfSense <1@users.surfsense>"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store(tmp_path, workspace_id) -> KnowledgeStore:
|
||||
# Virgin store: the first transaction must bootstrap it.
|
||||
engine = GitContentEngine(
|
||||
tmp_path / workspace_id, tmp_path / ".working_copies" / workspace_id
|
||||
)
|
||||
return KnowledgeStore(workspace_id, engine)
|
||||
|
||||
|
||||
async def test_transaction_records_one_revision(store):
|
||||
async with store.transaction(message="add note", author=AUTHOR) as tx:
|
||||
tx.write("documents/note.xml", b"hello")
|
||||
|
||||
assert tx.revision is not None
|
||||
assert await store.get_current_revision() == tx.revision
|
||||
assert await store.read_as_of(tx.revision, "documents/note.xml") == b"hello"
|
||||
|
||||
|
||||
async def test_failed_transaction_records_nothing(store):
|
||||
with pytest.raises(RuntimeError):
|
||||
async with store.transaction(message="doomed", author=AUTHOR) as tx:
|
||||
tx.write("documents/note.xml", b"hello")
|
||||
raise RuntimeError("boom")
|
||||
|
||||
assert await store.get_current_revision() is None
|
||||
|
||||
|
||||
async def test_transaction_fails_while_another_writer_holds_the_workspace(
|
||||
store, workspace_id, short_lock_wait
|
||||
):
|
||||
async with workspace_write_lock(workspace_id):
|
||||
with pytest.raises(KnowledgeStoreLockError):
|
||||
async with store.transaction(message="blocked", author=AUTHOR) as tx:
|
||||
tx.write("documents/note.xml", b"hello")
|
||||
|
||||
assert await store.get_current_revision() is None
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
"""workspace_write_lock over real Redis: one writer per workspace, fail over race."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
import app.knowledge_store.write_lock as write_lock
|
||||
from app.knowledge_store.write_lock import (
|
||||
KnowledgeStoreLockError,
|
||||
workspace_write_lock,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
async def test_serializes_writers_of_one_workspace(workspace_id, short_lock_wait):
|
||||
async with workspace_write_lock(workspace_id):
|
||||
with pytest.raises(KnowledgeStoreLockError):
|
||||
async with workspace_write_lock(workspace_id):
|
||||
pass
|
||||
|
||||
|
||||
async def test_workspaces_do_not_contend(workspace_id):
|
||||
async with workspace_write_lock(workspace_id):
|
||||
entered = False
|
||||
async with workspace_write_lock(f"{workspace_id}-other"):
|
||||
entered = True
|
||||
assert entered
|
||||
|
||||
|
||||
async def test_lock_is_released_on_scope_exit(workspace_id, short_lock_wait):
|
||||
async with workspace_write_lock(workspace_id):
|
||||
pass
|
||||
# A second writer succeeds because the first hold was released.
|
||||
async with workspace_write_lock(workspace_id):
|
||||
pass
|
||||
|
||||
|
||||
async def test_lock_is_released_when_the_scope_raises(workspace_id, short_lock_wait):
|
||||
with pytest.raises(RuntimeError):
|
||||
async with workspace_write_lock(workspace_id):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
async with workspace_write_lock(workspace_id):
|
||||
pass
|
||||
|
||||
|
||||
async def test_hold_outliving_the_ttl_fails_loudly(workspace_id, monkeypatch):
|
||||
monkeypatch.setattr(write_lock, "LOCK_TTL_SECONDS", 0.1)
|
||||
with pytest.raises(KnowledgeStoreLockError, match="expired mid-block"):
|
||||
async with workspace_write_lock(workspace_id):
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
|
||||
async def test_scope_error_is_not_masked_by_an_expired_hold(workspace_id, monkeypatch):
|
||||
monkeypatch.setattr(write_lock, "LOCK_TTL_SECONDS", 0.1)
|
||||
with pytest.raises(RuntimeError, match="boom"):
|
||||
async with workspace_write_lock(workspace_id):
|
||||
await asyncio.sleep(0.3)
|
||||
raise RuntimeError("boom")
|
||||
|
||||
|
||||
def test_a_second_lock_on_a_new_event_loop_still_works(workspace_id):
|
||||
"""Celery runs every task on its own loop, which is why the client cannot be
|
||||
cached: connections bound to a closed loop failed inside ``acquire``, after
|
||||
redis had set the key — leaking a lock nobody held for its whole TTL. Sync,
|
||||
so the loops here are the only ones in play."""
|
||||
|
||||
async def take_the_lock():
|
||||
async with workspace_write_lock(workspace_id):
|
||||
pass
|
||||
|
||||
for _ in range(2):
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
loop.run_until_complete(take_the_lock())
|
||||
finally:
|
||||
loop.close()
|
||||
|
|
@ -1,114 +0,0 @@
|
|||
"""One writer per document's chunks, once a workspace is git-backed.
|
||||
|
||||
The editor's reindex task re-chunks from Postgres ``source_markdown`` and titles
|
||||
the document from its first heading; the store indexer re-chunks from git and
|
||||
titles it from the filename. Both running means the title flips on every save and
|
||||
two writers reconcile the same chunk rows, so the loser's work is silently lost.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
|
||||
import app.tasks.celery_tasks.document_reindex_tasks as reindex_tasks
|
||||
from app.config import config as app_config
|
||||
from app.db import Document, DocumentStatus, DocumentType
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def celery_session_on_test_connection(db_session, monkeypatch):
|
||||
"""Point the task's own session maker at the test transaction."""
|
||||
maker = async_sessionmaker(
|
||||
bind=db_session.bind,
|
||||
expire_on_commit=False,
|
||||
join_transaction_mode="create_savepoint",
|
||||
)
|
||||
monkeypatch.setattr(reindex_tasks, "get_celery_session_maker", lambda: maker)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def reindex_spy(monkeypatch):
|
||||
calls: list[int] = []
|
||||
|
||||
async def _spy(self, *, document):
|
||||
calls.append(document.id)
|
||||
|
||||
monkeypatch.setattr(reindex_tasks.UploadDocumentAdapter, "reindex", _spy)
|
||||
return calls
|
||||
|
||||
|
||||
async def make_document(session, workspace_id, user_id) -> Document:
|
||||
document = Document(
|
||||
title="Editable",
|
||||
document_type=DocumentType.NOTE,
|
||||
document_metadata={},
|
||||
content="# Editable",
|
||||
content_hash=f"hash-{uuid.uuid4().hex}",
|
||||
unique_identifier_hash=f"unique-{uuid.uuid4().hex}",
|
||||
source_markdown="# Editable\n\nBody.",
|
||||
workspace_id=workspace_id,
|
||||
created_by_id=user_id,
|
||||
status=DocumentStatus.ready(),
|
||||
)
|
||||
session.add(document)
|
||||
await session.commit()
|
||||
return document
|
||||
|
||||
|
||||
async def test_the_editor_reindex_is_skipped_for_a_git_backed_workspace(
|
||||
db_session,
|
||||
db_workspace,
|
||||
db_user,
|
||||
monkeypatch,
|
||||
celery_session_on_test_connection,
|
||||
reindex_spy,
|
||||
workspace_flip,
|
||||
):
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
workspace_flip(True)
|
||||
document = await make_document(db_session, db_workspace.id, db_user.id)
|
||||
|
||||
await reindex_tasks._reindex_document(document.id, str(db_user.id))
|
||||
|
||||
assert reindex_spy == []
|
||||
|
||||
|
||||
async def test_the_editor_reindex_still_runs_for_an_unflipped_workspace(
|
||||
db_session,
|
||||
db_workspace,
|
||||
db_user,
|
||||
monkeypatch,
|
||||
celery_session_on_test_connection,
|
||||
reindex_spy,
|
||||
workspace_flip,
|
||||
):
|
||||
"""Global flag on, workspace not flipped: Postgres is still the write model."""
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
workspace_flip(False)
|
||||
document = await make_document(db_session, db_workspace.id, db_user.id)
|
||||
|
||||
await reindex_tasks._reindex_document(document.id, str(db_user.id))
|
||||
|
||||
assert reindex_spy == [document.id]
|
||||
|
||||
|
||||
async def test_the_editor_reindex_still_runs_without_the_store(
|
||||
db_session,
|
||||
db_workspace,
|
||||
db_user,
|
||||
monkeypatch,
|
||||
celery_session_on_test_connection,
|
||||
reindex_spy,
|
||||
):
|
||||
"""The guard must be conditional, not a quiet disabling of the legacy path."""
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", False)
|
||||
document = await make_document(db_session, db_workspace.id, db_user.id)
|
||||
|
||||
await reindex_tasks._reindex_document(document.id, str(db_user.id))
|
||||
|
||||
assert reindex_spy == [document.id]
|
||||
|
|
@ -32,96 +32,6 @@ async def db_document(
|
|||
return doc
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def client(db_session: AsyncSession, db_user: User):
|
||||
"""httpx over ASGI, bound to the transactional session and the owner."""
|
||||
import httpx
|
||||
from httpx import ASGITransport
|
||||
|
||||
from app.app import app
|
||||
from app.auth.context import AuthContext
|
||||
from app.db import get_async_session
|
||||
from app.users import get_auth_context
|
||||
|
||||
async def override_session():
|
||||
yield db_session
|
||||
|
||||
async def override_auth() -> AuthContext:
|
||||
return AuthContext.session(db_user)
|
||||
|
||||
previous = app.dependency_overrides.copy()
|
||||
app.dependency_overrides[get_async_session] = override_session
|
||||
app.dependency_overrides[get_auth_context] = override_auth
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
transport=ASGITransport(app=app), base_url="http://test", timeout=30.0
|
||||
) as c:
|
||||
yield c
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
app.dependency_overrides.update(previous)
|
||||
|
||||
|
||||
async def test_restore_is_refused_for_a_git_backed_workspace(
|
||||
client, db_session, db_document, monkeypatch, workspace_flip
|
||||
):
|
||||
"""Restore rewrites content behind git's back, so it cannot be allowed.
|
||||
|
||||
It is also the only writer of ``DocumentVersion`` rows on the save path, so
|
||||
this 409 is what makes "a save in a flagged workspace creates no version
|
||||
rows" true rather than merely intended.
|
||||
"""
|
||||
from app.config import config as app_config
|
||||
from app.utils.document_versioning import create_version_snapshot
|
||||
|
||||
await create_version_snapshot(db_session, db_document)
|
||||
await db_session.commit()
|
||||
before = await _version_count(db_session, db_document.id)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
workspace_flip(True)
|
||||
|
||||
response = await client.post(
|
||||
f"/api/v1/documents/{db_document.id}/versions/1/restore"
|
||||
)
|
||||
|
||||
assert response.status_code == 409
|
||||
assert await _version_count(db_session, db_document.id) == before
|
||||
|
||||
|
||||
async def test_restore_still_works_for_an_unflipped_workspace(
|
||||
client, db_session, db_document, monkeypatch, workspace_flip
|
||||
):
|
||||
"""The 409 is per workspace, not per deployment.
|
||||
|
||||
With the global flag on but this workspace still on Postgres, restore is the
|
||||
only way back — a guard keyed on the global flag would take it away from
|
||||
every workspace on the fleet the moment the flag is turned on.
|
||||
"""
|
||||
from app.config import config as app_config
|
||||
from app.utils.document_versioning import create_version_snapshot
|
||||
|
||||
# Past the 30-minute window, so restore's own pre-restore snapshot adds a
|
||||
# version instead of overwriting the one being restored.
|
||||
t0 = datetime(2025, 1, 1, 12, 0, 0, tzinfo=UTC)
|
||||
monkeypatch.setattr("app.utils.document_versioning._now", lambda: t0)
|
||||
await create_version_snapshot(db_session, db_document)
|
||||
db_document.source_markdown = "# Test\n\nLater content."
|
||||
db_document.content_hash = "def456"
|
||||
await db_session.commit()
|
||||
later = t0 + timedelta(minutes=31)
|
||||
monkeypatch.setattr("app.utils.document_versioning._now", lambda: later)
|
||||
monkeypatch.setattr(app_config, "KNOWLEDGE_STORE_ENABLED", True)
|
||||
workspace_flip(False)
|
||||
|
||||
response = await client.post(
|
||||
f"/api/v1/documents/{db_document.id}/versions/1/restore"
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
await db_session.refresh(db_document)
|
||||
assert db_document.source_markdown == "# Test\n\nOriginal content."
|
||||
|
||||
|
||||
async def _version_count(session: AsyncSession, document_id: int) -> int:
|
||||
result = await session.execute(
|
||||
select(func.count())
|
||||
|
|
|
|||
|
|
@ -118,9 +118,7 @@ class TestResolveMentions:
|
|||
document_type="EXTENSION",
|
||||
kind="doc",
|
||||
)
|
||||
doc_row = SimpleNamespace(
|
||||
id=42, title="Notes", folder_id=None, document_metadata=None
|
||||
)
|
||||
doc_row = SimpleNamespace(id=42, title="Notes", folder_id=None)
|
||||
|
||||
async def fake_build_index(_session, _ssid):
|
||||
return PathIndex()
|
||||
|
|
@ -222,10 +220,8 @@ class TestResolveMentions:
|
|||
id=2, title="A long one", document_type="EXTENSION", kind="doc"
|
||||
)
|
||||
rows = [
|
||||
SimpleNamespace(id=1, title="A", folder_id=None, document_metadata=None),
|
||||
SimpleNamespace(
|
||||
id=2, title="A long one", folder_id=None, document_metadata=None
|
||||
),
|
||||
SimpleNamespace(id=1, title="A", folder_id=None),
|
||||
SimpleNamespace(id=2, title="A long one", folder_id=None),
|
||||
]
|
||||
|
||||
async def fake_build_index(_session, _ssid):
|
||||
|
|
@ -253,9 +249,7 @@ class TestResolveMentions:
|
|||
# ``mentioned_document_ids`` (the legacy parallel array) must
|
||||
# still resolve when no chip metadata is available — covers
|
||||
# callers that haven't migrated to the discriminated chip list.
|
||||
doc_row = SimpleNamespace(
|
||||
id=7, title="Legacy", folder_id=None, document_metadata=None
|
||||
)
|
||||
doc_row = SimpleNamespace(id=7, title="Legacy", folder_id=None)
|
||||
|
||||
async def fake_build_index(_session, _ssid):
|
||||
return PathIndex()
|
||||
|
|
|
|||
|
|
@ -15,7 +15,6 @@ from app.agents.chat.runtime.path_resolver import (
|
|||
parse_documents_path,
|
||||
safe_filename,
|
||||
safe_folder_segment,
|
||||
to_store_path,
|
||||
virtual_path_to_doc,
|
||||
)
|
||||
|
||||
|
|
@ -44,25 +43,6 @@ class TestSafeFolderSegment:
|
|||
assert safe_folder_segment("") == "folder"
|
||||
|
||||
|
||||
class TestToStorePath:
|
||||
def test_keeps_the_documents_root(self):
|
||||
assert (
|
||||
to_store_path("/documents/Notes/Meeting.xml")
|
||||
== "documents/Notes/Meeting.xml"
|
||||
)
|
||||
|
||||
def test_root_itself_maps_to_the_documents_dir(self):
|
||||
assert to_store_path(DOCUMENTS_ROOT) == "documents"
|
||||
|
||||
def test_rejects_a_foreign_namespace(self):
|
||||
with pytest.raises(ValueError, match="/documents"):
|
||||
to_store_path("/documentsimposter/Meeting.xml")
|
||||
|
||||
def test_rejects_a_relative_path(self):
|
||||
with pytest.raises(ValueError, match="/documents"):
|
||||
to_store_path("Notes/Meeting.xml")
|
||||
|
||||
|
||||
class TestParseDocIdSuffix:
|
||||
def test_parses_suffix(self):
|
||||
stem, doc_id = parse_doc_id_suffix("My Doc (42).xml")
|
||||
|
|
|
|||
|
|
@ -2,42 +2,22 @@
|
|||
|
||||
The reconciler decides which rows (and embeddings) survive an edit, which texts
|
||||
must be embedded, and which rows go away -- purely from content, no DB.
|
||||
|
||||
Fixtures lay one chunk out per line, so chunk ``i`` sits on line ``i + 1``. That
|
||||
keeps line ranges in step with positions unless a test deliberately moves them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.indexing_pipeline.chunk_reconciler import (
|
||||
ExistingChunk,
|
||||
PendingChunk,
|
||||
ReusedChunk,
|
||||
reconcile,
|
||||
)
|
||||
from app.indexing_pipeline.document_chunker import LineChunk
|
||||
from app.indexing_pipeline.chunk_reconciler import ExistingChunk, reconcile
|
||||
|
||||
|
||||
def _existing(*contents: str) -> list[ExistingChunk]:
|
||||
return [
|
||||
ExistingChunk(
|
||||
id=i + 1, content=text, position=i, start_line=i + 1, end_line=i + 1
|
||||
)
|
||||
for i, text in enumerate(contents)
|
||||
]
|
||||
|
||||
|
||||
def _new(*contents: str) -> list[LineChunk]:
|
||||
return [
|
||||
LineChunk(text=text, start_line=i + 1, end_line=i + 1)
|
||||
ExistingChunk(id=i + 1, content=text, position=i)
|
||||
for i, text in enumerate(contents)
|
||||
]
|
||||
|
||||
|
||||
def test_identical_content_keeps_every_row_untouched():
|
||||
plan = reconcile(
|
||||
_existing("alpha", "beta", "gamma"), _new("alpha", "beta", "gamma")
|
||||
)
|
||||
plan = reconcile(_existing("alpha", "beta", "gamma"), ["alpha", "beta", "gamma"])
|
||||
|
||||
assert plan.to_embed == []
|
||||
assert plan.to_delete == []
|
||||
|
|
@ -45,109 +25,70 @@ def test_identical_content_keeps_every_row_untouched():
|
|||
|
||||
|
||||
def test_head_insert_embeds_only_the_new_chunk_and_shifts_the_rest():
|
||||
plan = reconcile(_existing("alpha", "beta"), _new("intro", "alpha", "beta"))
|
||||
plan = reconcile(_existing("alpha", "beta"), ["intro", "alpha", "beta"])
|
||||
|
||||
assert plan.to_embed == [PendingChunk(0, "intro", 1, 1)]
|
||||
assert plan.to_embed == [(0, "intro")]
|
||||
assert plan.to_delete == []
|
||||
# alpha: position 0 -> 1, beta: 1 -> 2; embeddings untouched.
|
||||
assert plan.reused == [ReusedChunk(1, 1, 2, 2), ReusedChunk(2, 2, 3, 3)]
|
||||
assert plan.reused == [(1, 1), (2, 2)]
|
||||
|
||||
|
||||
def test_middle_edit_swaps_exactly_one_chunk():
|
||||
plan = reconcile(
|
||||
_existing("alpha", "beta", "gamma"), _new("alpha", "beta EDITED", "gamma")
|
||||
_existing("alpha", "beta", "gamma"), ["alpha", "beta EDITED", "gamma"]
|
||||
)
|
||||
|
||||
assert plan.to_embed == [PendingChunk(1, "beta EDITED", 2, 2)]
|
||||
assert plan.to_embed == [(1, "beta EDITED")]
|
||||
assert plan.to_delete == [2]
|
||||
# Neighbours did not move, so no writes at all.
|
||||
# Neighbours did not move, so no position writes at all.
|
||||
assert plan.reused == []
|
||||
|
||||
|
||||
def test_unchanged_text_that_moved_lines_is_still_written_back():
|
||||
"""An edit above a chunk shifts its lines without moving its position.
|
||||
|
||||
The embedding survives, but the stored line range is now wrong -- so the row
|
||||
has to be updated even though nothing about it looks like it moved.
|
||||
"""
|
||||
existing = _existing("alpha", "beta")
|
||||
new_chunks = [
|
||||
LineChunk(text="alpha GREW", start_line=1, end_line=2),
|
||||
LineChunk(text="beta", start_line=3, end_line=3),
|
||||
]
|
||||
|
||||
plan = reconcile(existing, new_chunks)
|
||||
|
||||
assert plan.to_embed == [PendingChunk(0, "alpha GREW", 1, 2)]
|
||||
assert plan.to_delete == [1]
|
||||
assert plan.reused == [ReusedChunk(2, 1, 3, 3)]
|
||||
|
||||
|
||||
def test_removed_chunk_is_deleted_and_followers_shift_up():
|
||||
plan = reconcile(_existing("alpha", "beta", "gamma"), _new("alpha", "gamma"))
|
||||
plan = reconcile(_existing("alpha", "beta", "gamma"), ["alpha", "gamma"])
|
||||
|
||||
assert plan.to_embed == []
|
||||
assert plan.to_delete == [2]
|
||||
assert plan.reused == [ReusedChunk(3, 1, 2, 2)]
|
||||
assert plan.reused == [(3, 1)]
|
||||
|
||||
|
||||
def test_duplicate_texts_pair_up_one_to_one():
|
||||
# Two identical boilerplate chunks, only one survives the edit: exactly one
|
||||
# row is kept and exactly one is deleted -- never both kept or both dropped.
|
||||
plan = reconcile(_existing("boiler", "boiler", "body"), _new("boiler", "body"))
|
||||
plan = reconcile(_existing("boiler", "boiler", "body"), ["boiler", "body"])
|
||||
|
||||
assert plan.to_embed == []
|
||||
assert plan.to_delete == [2]
|
||||
assert plan.reused == [ReusedChunk(3, 1, 2, 2)]
|
||||
assert plan.reused == [(3, 1)]
|
||||
|
||||
|
||||
def test_duplicate_growth_embeds_only_the_extra_copy():
|
||||
plan = reconcile(_existing("boiler", "body"), _new("boiler", "boiler", "body"))
|
||||
plan = reconcile(_existing("boiler", "body"), ["boiler", "boiler", "body"])
|
||||
|
||||
assert plan.to_embed == [PendingChunk(1, "boiler", 2, 2)]
|
||||
assert plan.to_embed == [(1, "boiler")]
|
||||
assert plan.to_delete == []
|
||||
assert plan.reused == [ReusedChunk(2, 2, 3, 3)]
|
||||
assert plan.reused == [(2, 2)]
|
||||
|
||||
|
||||
def test_reorder_becomes_position_updates_with_no_embedding():
|
||||
plan = reconcile(_existing("alpha", "beta"), _new("beta", "alpha"))
|
||||
plan = reconcile(_existing("alpha", "beta"), ["beta", "alpha"])
|
||||
|
||||
assert plan.to_embed == []
|
||||
assert plan.to_delete == []
|
||||
assert sorted(plan.reused, key=lambda r: r.id) == [
|
||||
ReusedChunk(1, 1, 2, 2),
|
||||
ReusedChunk(2, 0, 1, 1),
|
||||
]
|
||||
assert sorted(plan.reused) == [(1, 1), (2, 0)]
|
||||
|
||||
|
||||
def test_full_rewrite_replaces_everything():
|
||||
plan = reconcile(_existing("alpha", "beta"), _new("new one", "new two"))
|
||||
plan = reconcile(_existing("alpha", "beta"), ["new one", "new two"])
|
||||
|
||||
assert plan.to_embed == [
|
||||
PendingChunk(0, "new one", 1, 1),
|
||||
PendingChunk(1, "new two", 2, 2),
|
||||
]
|
||||
assert plan.to_embed == [(0, "new one"), (1, "new two")]
|
||||
assert sorted(plan.to_delete) == [1, 2]
|
||||
assert plan.reused == []
|
||||
|
||||
|
||||
def test_no_existing_chunks_embeds_all():
|
||||
plan = reconcile([], _new("alpha", "beta"))
|
||||
plan = reconcile([], ["alpha", "beta"])
|
||||
|
||||
assert plan.to_embed == [
|
||||
PendingChunk(0, "alpha", 1, 1),
|
||||
PendingChunk(1, "beta", 2, 2),
|
||||
]
|
||||
assert plan.to_embed == [(0, "alpha"), (1, "beta")]
|
||||
assert plan.to_delete == []
|
||||
assert plan.reused == []
|
||||
|
||||
|
||||
def test_spanless_legacy_row_is_backfilled():
|
||||
"""Rows written before spans existed carry NULL; matching them rewrites them."""
|
||||
existing = [ExistingChunk(id=7, content="alpha", position=0)]
|
||||
|
||||
plan = reconcile(existing, _new("alpha"))
|
||||
|
||||
assert plan.to_embed == []
|
||||
assert plan.to_delete == []
|
||||
assert plan.reused == [ReusedChunk(7, 0, 1, 1)]
|
||||
|
|
|
|||
|
|
@ -1,88 +0,0 @@
|
|||
"""Line spans must name the exact slice a chunk was cut from.
|
||||
|
||||
Spans are what a search excerpt renders line numbers from, so an off-by-one here
|
||||
points a reader at the wrong line. Every case asserts against the document's own
|
||||
lines rather than against hardcoded numbers, so the assertions stay true if the
|
||||
fixture text is edited.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.indexing_pipeline.document_chunker import attach_line_spans
|
||||
|
||||
|
||||
def slice_of(text: str, span) -> str:
|
||||
"""The document lines a span claims, as the reader would see them."""
|
||||
lines = text.split("\n")
|
||||
return "\n".join(lines[span.start_line - 1 : span.end_line])
|
||||
|
||||
|
||||
def test_a_single_line_chunk_claims_exactly_its_line():
|
||||
text = "alpha\nbravo\ncharlie"
|
||||
|
||||
spans = attach_line_spans(text, ["bravo"])
|
||||
|
||||
assert (spans[0].start_line, spans[0].end_line) == (2, 2)
|
||||
|
||||
|
||||
def test_every_chunk_span_slices_back_to_its_own_text():
|
||||
text = "# Title\n\nFirst paragraph.\n\nSecond paragraph.\n"
|
||||
chunks = ["# Title", "First paragraph.", "Second paragraph."]
|
||||
|
||||
spans = attach_line_spans(text, chunks)
|
||||
|
||||
for span in spans:
|
||||
assert span.text in slice_of(text, span)
|
||||
|
||||
|
||||
def test_a_multi_line_chunk_spans_from_its_first_line_to_its_last():
|
||||
text = "intro\n\nline one\nline two\nline three\n\noutro"
|
||||
chunk = "line one\nline two\nline three"
|
||||
|
||||
span = attach_line_spans(text, [chunk])[0]
|
||||
|
||||
assert (span.start_line, span.end_line) == (3, 5)
|
||||
assert slice_of(text, span) == chunk
|
||||
|
||||
|
||||
def test_a_chunk_ending_in_a_newline_does_not_claim_the_next_line():
|
||||
"""The trailing newline belongs to the chunk's last line, not the one after."""
|
||||
text = "first\nsecond\nthird"
|
||||
|
||||
span = attach_line_spans(text, ["first\nsecond\n"])[0]
|
||||
|
||||
assert (span.start_line, span.end_line) == (1, 2)
|
||||
|
||||
|
||||
def test_repeated_text_resolves_to_each_occurrence_in_order():
|
||||
"""A whole-document search would pin both chunks to line 1; the cursor cannot."""
|
||||
text = "duplicate\nmiddle\nduplicate\n"
|
||||
|
||||
spans = attach_line_spans(text, ["duplicate", "duplicate"])
|
||||
|
||||
assert [(s.start_line, s.end_line) for s in spans] == [(1, 1), (3, 3)]
|
||||
|
||||
|
||||
def test_a_markdown_table_kept_whole_spans_all_of_its_rows():
|
||||
"""The hybrid chunker emits a table as one chunk and strips it as it goes."""
|
||||
text = "Before the table.\n\n| a | b |\n| - | - |\n| 1 | 2 |\n\nAfter the table.\n"
|
||||
table = "| a | b |\n| - | - |\n| 1 | 2 |"
|
||||
|
||||
spans = attach_line_spans(text, ["Before the table.", table, "After the table."])
|
||||
|
||||
assert [(s.start_line, s.end_line) for s in spans] == [(1, 1), (3, 5), (7, 7)]
|
||||
assert slice_of(text, spans[1]) == table
|
||||
|
||||
|
||||
def test_a_chunk_that_is_not_in_the_document_falls_back_to_the_cursor():
|
||||
"""A chunker that rewrites text must degrade to a plausible line, not crash."""
|
||||
text = "alpha\nbravo\ncharlie\n"
|
||||
|
||||
spans = attach_line_spans(text, ["alpha", "REWRITTEN"])
|
||||
|
||||
assert spans[1].start_line >= spans[0].start_line
|
||||
assert spans[1].start_line <= text.count("\n") + 1
|
||||
|
||||
|
||||
def test_no_chunks_yields_no_spans():
|
||||
assert attach_line_spans("anything", []) == []
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue