SurfSense/docs/architecture/search.md

Search

retrieve() finds the passages in a workspace that best answer a query. Two legs widen recall, a keyword match and a nearest-neighbour search over embeddings, and a weighted blend of the two decides the order, meaning counting for rather more than words. It runs in the calling process against the same SQLite file as everything else. Chat calls it, and so does Studio's grounding (studio.md, Grounding).

Code: shared/search.py, shared/tokenizer.py, modules/embedding/ Embedder: embedding Decisions: ADR 0006, ADR 0007, ADR 0036, ADR 0031, ADR 0032, ADR 0033

Interface

retrieve(session, workspace_id, query, top_k=5, document_ids=None, scope=None) -> list[Hit]

• A Hit carries chunk_id, document_id, the document's title, the passage content, start_line, end_line and score, which is 1 minus the cosine distance. Hits come back best first.
• The workspace is the scope; chunks are what is searched.
• scope is a source scope the server resolved (chat.md): its ready ids narrow the search, and one with none returns nothing. It wins over document_ids.
• document_ids narrows the scope to those documents. None searches the whole workspace, and an empty list returns nothing. Plugins and the agent pass it.
• Either list is bound once as JSON and read with json_each, so its length costs no bind parameters.
• A blank query returns nothing before anything is embedded.

How it ranks

1. Embed the query with the active embedding index's model, the one ingest wrote with, adding the spec's query prefix if it has one (bge-small has none). The import is lazy, so onnxruntime and the model load on the first query rather than when the API starts. With no index chosen yet, retrieve() raises the same embedding_not_chosen error the routes answer 409 with.
2. Keyword leg. The query is split into terms by the index's own tokenizer, each quoted against FTS5's query grammar and joined with OR to keep recall wide. chunks_fts is matched, joined to chunks and documents for the workspace and document filters, and the best 20 by BM25 are kept. Each candidate then scores the fraction of the query's distinct terms it matched, one further lookup per term.
3. Vector leg. A sqlite-vec nearest-neighbour search over the active index's vector table, chunk_vectors, takes the 20 closest chunks of the workspace or the scope. It sits in its own CTE, as vec0 requires, with rowid IN (the chunks of the workspace or scope) beside MATCH and k: sqlite-vec 0.1.9 applies that filter before it takes k, so another workspace or an unticked folder can never crowd a scope's chunks out. It proposes those chunks and does not score them.
4. Blend. w × semantic + (1 − w) × keyword over the union, where w is the active spec's semantic_weight, 0.65 for bge-small, cut to top_k with cosine breaking a tie. Semantic strength is each candidate's own cosine similarity, floored at zero, measured here for the whole union rather than taken from the vector leg (ADR 0033). A candidate the keyword leg is missing contributes nothing from it, that absence being a measurement.

Both legs decide the order, and both score on a scale that means the same thing from one query to the next, so a leg with nothing to say adds nothing. That is what keeps a keyword match honest: a chunk matching one term of five is weak whatever BM25 says about it, so no stopword list is needed. It holds because a term is a word: shared/tokenizer.py states the one rule that splits both the index and the question, and keeps a combining mark inside its word so Devanagari is not cut into letters (ADR 0032). A paraphrase that shares no words with its passage still arrives through the vector leg. There is no reranker.

The two legs are asymmetric on purpose. A chunk the keyword leg omits matched none of the query's terms, so zero is what it measured; a chunk the vector leg omits merely sits outside the nearest 20 of its scope, which says nothing about it.

Hit.score is cosine similarity, so it says how close a passage is, not where it sits.

Index

chunks_fts is an FTS5 table with external content over chunks, declared with the tokenizer shared/tokenizer.py names, and chunk_vectors a vec0 table keyed by chunk id. Triggers on chunks keep the keyword index in step, including on cascade, and ingest writes the vectors (data-model.md, documents.md). Because both indexes key on chunk ids, a hit is a chunk row, and the citation panel can load that chunk's neighbours by position.

Tests

tests/integration/search/test_retrieve.py covers a keyword query, a paraphrase found through meaning, a Hindi question against three notes one word apart, a workspace whose neighbour holds every one of the 20 nearest chunks, a paraphrase in my workspace past such a neighbour, a one-document scope among many nearer chunks, the document and lines on a hit, scoping to selected documents and to the workspace, and the empty workspace, query, selection and scope. tests/integration/chunks/test_search_index.py covers the triggers, the refusal of a vector of the wrong width, and that the index and a question split text the same way — the migration and shared/tokenizer.py state the tokenizer separately, and a drift between them scores a question against terms the index never held.

Ranking itself is measured rather than asserted, by scripts/run_retrieval_eval.py: it indexes a fixed corpus through the real ingest pipeline and records where each query's answering passage landed. Failures that need a library-sized corpus, a bare identifier among near-duplicate manuals being the one that drove ADR 0031, do not reproduce at the handful of documents an integration test builds.

Two things it cannot currently show. LIMIT-small is 200 of its 266 queries, so the all row is mostly LIMIT and the per-slice rows are what to read. And the four same-language slices sit at 100% for every model and weight tried — saturated, so they can neither fail nor improve, which makes them blind to a regression an embedder change would cause (retrieval proposal).

Known gaps

• A question in one language does not find its answer in another. Cosine alone puts the answering passage in the top 5 for 1 of 8 such queries, and the blend for 2 of 8: bge-small is English-only, so no ranking recovers it. Same-language retrieval is unaffected, at 100% in all four non-English slices. A multilingual embedder is the fix, designed in the retrieval proposal.
• FTS5's tokenizer keeps a Japanese or Chinese clause as one token, so the keyword leg finds nothing in those scripts and the blend runs on meaning alone there. trigram would segment them at the cost of what BM25 means everywhere else (ADR 0032).