mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-10-19 15:44:27 +00:00
84c7ffad18
The embedding model is now chosen in the last onboarding step and fixed for the whole library. Skipping keeps bge-small, which ships with the app and can't be deleted. Changing the model later is not built. - An embedding_indexes row records the model's spec and its vector table, and each document records which index embedded it. Migration 0021 stamps existing libraries with bge-small. Routes that queue embedding work return 409 embedding_not_chosen until a model is chosen. - Ingest, Studio and search embed through the active index; the old worker embedding module is removed. Workers wait for the schema, and documents interrupted when the app closed are marked failed. - Curated embedders (bge-small, granite-97m, granite-311m) live in the local manifest, run by a new onnxruntime engine. EMBEDDING is a catalog type, never a selection slot. - Hugging Face search finds ONNX embedders. Each download is checked: a probe sets its width, and a ten-question search check must pass. Small tokenizer.json files that Hugging Face lists without a checksum are hashed at the pinned commit. Results use the GGUF search's shapes. - Onboarding renders the embedding step with the same component and shared list as the other steps; only its choice hook differs. Use marks the choice and Finish sends it. Bundled models have no Delete. - Settings has a read-only Embedding section. - ADRs 0036 and 0037, a new embedding architecture doc, and updates to the docs it touches. New strings are translated into every language.
60 lines
1.5 KiB
Python
60 lines
1.5 KiB
Python
"""Everything that decides what a vector means.
|
|
|
|
An index stores a snapshot of this, so a later edit to a manifest entry cannot
|
|
change what an existing index's vectors are.
|
|
"""
|
|
|
|
from enum import StrEnum
|
|
|
|
from pydantic import BaseModel, ConfigDict, Field
|
|
|
|
|
|
class Source(StrEnum):
|
|
CURATED = "curated"
|
|
HUGGINGFACE = "huggingface"
|
|
REMOTE = "remote"
|
|
|
|
|
|
class Identified(StrEnum):
|
|
"""How SurfSense knows the model is built to embed."""
|
|
|
|
MEASURED = "measured"
|
|
DECLARED = "declared"
|
|
INFERRED = "inferred"
|
|
UNVERIFIED = "unverified"
|
|
|
|
|
|
class Pooling(StrEnum):
|
|
CLS = "cls"
|
|
MEAN = "mean"
|
|
LAST = "last"
|
|
# The build pools inside its own graph and returns one vector per text.
|
|
IN_MODEL = "in_model"
|
|
|
|
|
|
class PinnedFile(BaseModel):
|
|
model_config = ConfigDict(extra="forbid", frozen=True)
|
|
|
|
path: str = Field(min_length=1)
|
|
sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
|
|
|
|
|
|
class EmbedderSpec(BaseModel):
|
|
model_config = ConfigDict(extra="forbid", frozen=True)
|
|
|
|
id: str = Field(min_length=1)
|
|
source: Source
|
|
identified: Identified
|
|
repo: str = Field(min_length=1)
|
|
revision: str = Field(pattern=r"^[0-9a-f]{40}$")
|
|
weights: PinnedFile
|
|
tokenizer: PinnedFile
|
|
dimension: int = Field(gt=0)
|
|
pooling: Pooling
|
|
normalize: bool
|
|
query_prefix: str = ""
|
|
document_prefix: str = ""
|
|
max_tokens: int = Field(gt=0)
|
|
semantic_weight: float = Field(gt=0, lt=1)
|
|
# Passages embedded at once; the batch, not the weights, sets peak memory.
|
|
batch: int = Field(default=32, gt=0)
|