mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-10-19 07:34:58 +00:00
84c7ffad18
The embedding model is now chosen in the last onboarding step and fixed for the whole library. Skipping keeps bge-small, which ships with the app and can't be deleted. Changing the model later is not built. - An embedding_indexes row records the model's spec and its vector table, and each document records which index embedded it. Migration 0021 stamps existing libraries with bge-small. Routes that queue embedding work return 409 embedding_not_chosen until a model is chosen. - Ingest, Studio and search embed through the active index; the old worker embedding module is removed. Workers wait for the schema, and documents interrupted when the app closed are marked failed. - Curated embedders (bge-small, granite-97m, granite-311m) live in the local manifest, run by a new onnxruntime engine. EMBEDDING is a catalog type, never a selection slot. - Hugging Face search finds ONNX embedders. Each download is checked: a probe sets its width, and a ten-question search check must pass. Small tokenizer.json files that Hugging Face lists without a checksum are hashed at the pinned commit. Results use the GGUF search's shapes. - Onboarding renders the embedding step with the same component and shared list as the other steps; only its choice hook differs. Use marks the choice and Finish sends it. Bundled models have no Delete. - Settings has a read-only Embedding section. - ADRs 0036 and 0037, a new embedding architecture doc, and updates to the docs it touches. New strings are translated into every language.
27 lines
1.1 KiB
Python
27 lines
1.1 KiB
Python
"""Whether a model SurfSense did not measure works, after its download."""
|
|
|
|
from modules.embedding.encoder import Purpose, embed, width
|
|
from modules.embedding.search_check import search_check
|
|
from modules.embedding.spec import EmbedderSpec
|
|
|
|
PROBE = "The scanner battery lasts about ten hours on a full charge."
|
|
|
|
|
|
def verify(spec: EmbedderSpec) -> tuple[int, str | None]:
|
|
"""Its real width, and why it is refused, or None. The probe's width wins
|
|
over the config's, which a model with a projection layer gets wrong."""
|
|
probed = spec.model_copy(update={"dimension": width(spec, PROBE)})
|
|
result = search_check(
|
|
lambda texts: _embed(probed, texts, Purpose.QUERY),
|
|
lambda texts: _embed(probed, texts, Purpose.DOCUMENT),
|
|
)
|
|
if not result.passed:
|
|
return probed.dimension, (
|
|
f"It found {result.first} of {result.asked} answers first; a search "
|
|
f"model has to find all {result.asked}."
|
|
)
|
|
return probed.dimension, None
|
|
|
|
|
|
def _embed(spec: EmbedderSpec, texts: list[str], purpose: Purpose) -> list[list[float]]:
|
|
return embed(spec, texts, purpose)
|