mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-11-16 21:37:13 +00:00
The embedding model is now chosen in the last onboarding step and fixed for the whole library. Skipping keeps bge-small, which ships with the app and can't be deleted. Changing the model later is not built. - An embedding_indexes row records the model's spec and its vector table, and each document records which index embedded it. Migration 0021 stamps existing libraries with bge-small. Routes that queue embedding work return 409 embedding_not_chosen until a model is chosen. - Ingest, Studio and search embed through the active index; the old worker embedding module is removed. Workers wait for the schema, and documents interrupted when the app closed are marked failed. - Curated embedders (bge-small, granite-97m, granite-311m) live in the local manifest, run by a new onnxruntime engine. EMBEDDING is a catalog type, never a selection slot. - Hugging Face search finds ONNX embedders. Each download is checked: a probe sets its width, and a ten-question search check must pass. Small tokenizer.json files that Hugging Face lists without a checksum are hashed at the pinned commit. Results use the GGUF search's shapes. - Onboarding renders the embedding step with the same component and shared list as the other steps; only its choice hook differs. Use marks the choice and Finish sends it. Bundled models have no Delete. - Settings has a read-only Embedding section. - ADRs 0036 and 0037, a new embedding architecture doc, and updates to the docs it touches. New strings are translated into every language.
27 lines
845 B
Python
27 lines
845 B
Python
"""The vec0 table an index's vectors live in.
|
|
|
|
Its name comes from a database row and is spliced into SQL, so it is checked
|
|
against a fixed shape first.
|
|
"""
|
|
|
|
import re
|
|
|
|
from sqlalchemy import Connection, text
|
|
|
|
_NAME = re.compile(r"^chunk_vectors(_[0-9]+)?$")
|
|
_DECLARED_WIDTH = re.compile(r"float\[(\d+)\]")
|
|
|
|
|
|
def checked(name: str) -> str:
|
|
if not _NAME.fullmatch(name):
|
|
raise ValueError(f"not a vector table name: {name!r}")
|
|
return name
|
|
|
|
|
|
def declared_width(connection: Connection, name: str) -> int | None:
|
|
"""The width the table was created with, or None if it does not exist."""
|
|
schema = connection.execute(
|
|
text("SELECT sql FROM sqlite_master WHERE name = :name"),
|
|
{"name": checked(name)},
|
|
).scalar_one_or_none()
|
|
return None if schema is None else int(_DECLARED_WIDTH.search(schema).group(1))
|