mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-10-19 15:44:27 +00:00
docs(proposals): curate embedders in the local manifest, add server adapters
Curated embedders join the shared local manifest as a fourth engine for ONNX embedding models. EMBEDDING becomes a catalog type that is never a selection slot, guarded in selectable_for(), choose_model() and the selected_models check constraint. The curated list is bge-small (the default), then granite 97M and 311M multilingual r2, then Harrier 270M. Each entry pins a generic int8 build, admitted only if its vectors match the original model's. The spec's pooling gains a value for builds that pool inside the model, and the manifest entry gains a batch size. As in Unsloth Studio, there is no size or memory cap: sizes are shown, the curated list stays short by choice, and a Hugging Face pick says what a larger model costs. A Hugging Face repo's ONNX file is chosen in a fixed order: generic int8, then full precision, else the repo is refused. Remote embedders are recognised first by asking the server. Ollama, LM Studio, TEI and OpenRouter each say what a model is through their own API. For TEI and OpenRouter the adapter also supplies the list of models, since TEI has no /models route and OpenRouter's leaves embedders out. Adapters reach past /v1 on the same host and fall back to older API versions. llama-server and vLLM report nothing about a model's kind, and they and Ollama return vectors from chat models, which is why the sanity check is needed. Machine-specific timings are removed from both proposals.
This commit is contained in:
parent
3d4d5f2df6
commit
67e07d9949
2 changed files with 87 additions and 20 deletions
|
|
@ -51,7 +51,7 @@ The section shows the old model as in use, the new one as building, progress as
|
|||
|
||||
## Cost
|
||||
|
||||
- Time: bge measured at about 8 minutes per 1,000 passages on a mid CPU ([retrieval](retrieval.md)); a larger model is slower.
|
||||
- Time: proportional to the size of the library and the speed of the new model.
|
||||
- Disk: both vector tables exist until the switch, so vector storage roughly doubles for the length of the change.
|
||||
- Memory: the ingest worker holds both models while new uploads write to both indexes.
|
||||
- Money: a change to a remote model is billed for every passage in the library, and uploads during the change embed twice.
|
||||
|
|
|
|||
|
|
@ -2,6 +2,11 @@
|
|||
status: proposed
|
||||
code:
|
||||
- surfsense_local/backend/modules/embedding/
|
||||
- surfsense_local/backend/modules/llm/catalog/local/
|
||||
- surfsense_local/backend/modules/llm/model_type.py
|
||||
- surfsense_local/backend/modules/llm/selectable.py
|
||||
- surfsense_local/backend/modules/llm/selection.py
|
||||
- surfsense_local/backend/scripts/local_manifest/
|
||||
- surfsense_local/backend/worker/ingestion/
|
||||
- surfsense_local/backend/shared/search.py
|
||||
- surfsense_local/backend/shared/migrations.py
|
||||
|
|
@ -31,7 +36,7 @@ Today one model, bge-small-en-v1.5, is hard-coded in [`embedding.py`](../../surf
|
|||
- **Onboarding is the only place to choose.** An install that finished onboarding before this ships is locked to bge.
|
||||
- **Stricter than the other model types.** Chat, image and audio admit a model nothing recognises, because the user sees it answer before trusting it ([`selectable.py`](../../surfsense_local/backend/modules/llm/selectable.py)). A wrong embedder never fails where the user can see it: it returns vectors, and search ranks badly. And the pick is permanent. So an embedder is refused unless evidence says it is one, and every pick SurfSense did not measure is tested before it is accepted.
|
||||
- **No fallback, ever.** If the locked model becomes unreachable, search and ingest stop and say why. Answering with bge would compare bge questions against another model's vectors, which returns nonsense rather than an error.
|
||||
- **The risk is accepted, not solved.** A pick that passes its checks and still ranks badly, or a remote model its provider retires, stays until [changing](embedding-model-change.md) ships. Onboarding says so in plain words before the user commits.
|
||||
- **The risk is accepted, not solved.** A pick that passes its checks and still ranks badly, or a remote model its provider retires, stays until [changing](embedding-model-change.md) ships. Onboarding says so in plain words before the user commits. A smaller way back, a blocking rebuild that pauses search and re-embeds every chunk, was considered and deferred with the rest of [changing](embedding-model-change.md).
|
||||
|
||||
## How a pick is admitted
|
||||
|
||||
|
|
@ -67,7 +72,7 @@ Every source produces the same `EmbedderSpec`, and the index stores a snapshot o
|
|||
| `repo`, `revision`, file hashes | local only: what to download, pinned, so a re-download is the same model |
|
||||
| `connection_id`, `model_id` | remote only: where to call |
|
||||
| `dimension` | the vector table's width, from the probe |
|
||||
| `pooling`, `normalize` | local only: the same model pooled another way is another space |
|
||||
| `pooling`, `normalize` | local only: the same model pooled another way is another space. `in_model` when the build pools inside its own graph and returns one vector per text, so the encoder does not pool twice |
|
||||
| `query_prefix`, `document_prefix` | asymmetric local models embed questions and passages differently |
|
||||
| `input_mode` | remote only: a provider's own query and document switch, where it has one |
|
||||
| `max_tokens` | a chunk past it is cut short |
|
||||
|
|
@ -76,22 +81,63 @@ Every source produces the same `EmbedderSpec`, and the index stores a snapshot o
|
|||
|
||||
## Curated
|
||||
|
||||
A manifest in `modules/embedding/`. Not the [local catalog manifest](../../surfsense_local/backend/modules/llm/catalog/local/manifest/models.json): its validator requires a bundled engine for each entry, its files are GGUF, and it feeds the model slots this design keeps the embedder out of. It pins files the same way, by revision and sha256.
|
||||
### The list
|
||||
|
||||
Ordered by preference, as every curated list is, each shown with its download size. All are ONNX, ungated and permissively licensed; numbers are from each model's card and the int8 build that would be pinned.
|
||||
|
||||
| Model | Licence | Params | Width | Languages | Int8 build | Pooling, prefix | Status |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| `BAAI/bge-small-en-v1.5` | MIT | 33M | 384 | English | bundled | CLS, none | the default |
|
||||
| `ibm-granite/granite-embedding-97m-multilingual-r2` | Apache-2.0 | 97M | 384 | 200+, 52 strong | 98 MB | CLS, none | first to add: same width as bge, runs on today's encoder |
|
||||
| `ibm-granite/granite-embedding-311m-multilingual-r2` | Apache-2.0 | 311M | 768 | 200+, 52 strong | 313 MB | CLS, none | for machines with more memory |
|
||||
| `microsoft/harrier-oss-v1-270m` | MIT | 270M | 640 | multilingual | 344 MB | in the model, instruction on queries | after the open questions below |
|
||||
|
||||
The cards do not report the same benchmark: granite gives multilingual retrieval (60.3 and 65.2), harrier the multilingual MTEB v2 average across task types (66.5). The retrieval eval decides, cross-language slice included.
|
||||
|
||||
Left out: harrier-oss-v1-0.6b (too slow and heavy for most machines), EmbeddingGemma-300m (gated), jina-embeddings v3 and v5 (non-commercial), Qwen3-Embedding-0.6B (behind harrier at the same size, and as heavy), bge-m3 (2.2 GB, older and weaker), the multilingual-e5 family (older), nomic-embed-text-v1.5 and mxbai-embed-large (English only), and anything with no ONNX build or a licence of its own.
|
||||
|
||||
An entry is admitted only if:
|
||||
|
||||
- it ships an ONNX file, is ungated and permissively licensed;
|
||||
- the retrieval eval measured its `semantic_weight`;
|
||||
- every chunk of the eval corpus, cut at `CHUNK_TOKENS`, fits its `max_tokens` in its own tokenizer, in every language (see *Chunking* below);
|
||||
- it is small enough to load once per embedding process: the API for questions, the ingest worker, and the Studio worker ([`persist.py`](../../surfsense_local/backend/worker/studio/shared/persist.py)).
|
||||
- its pinned build matches the original model (see *Builds* below).
|
||||
|
||||
Every curated pick is `measured`. The first candidate after bge is `ibm-granite/granite-embedding-97m-multilingual-r2`, with the risks [retrieval](retrieval.md) lists still open.
|
||||
There is no size or memory cap. As in [Unsloth Studio](https://github.com/unslothai/unsloth/tree/main/studio), the size is shown and the choice is the user's; the list stays short because SurfSense chooses what goes on it. A model too heavy for most machines is simply not listed.
|
||||
|
||||
### Builds
|
||||
|
||||
A curated entry pins a generic int8 build, not one tuned for a single CPU such as granite's own `model_quint8_avx2.onnx`. Most come from a conversion someone else made, so a build is pinned at a revision and admitted only if its vectors match the original model's on a sample of the eval corpus. A build that fails is replaced by an export of our own.
|
||||
|
||||
A build may pool inside its own graph and return one vector per text, as the harrier conversions do, or return every token's output and leave pooling to the encoder, as bge and granite do. The spec's `pooling` says which.
|
||||
|
||||
Every curated pick is `measured`.
|
||||
|
||||
### Where the list lives
|
||||
|
||||
In the same [local manifest](../../surfsense_local/backend/modules/llm/catalog/local/manifest/models.json) as chat, image and audio models, run by a fourth engine. It is then refreshed, pinned, licence-checked, downloaded, shown and deleted by the code that already does all of that for the other types.
|
||||
|
||||
- **An ONNX embedding engine** in the [engine registry](../../surfsense_local/backend/modules/llm/catalog/local/engines/registry.py), beside llama.cpp, sd.cpp and audio.cpp. Its entry block, `embedding`, holds what the spec needs and the header cannot give: `dimension`, `pooling`, `normalize`, `query_prefix`, `document_prefix`, `max_tokens` and `semantic_weight`, and `batch`, the passages embedded at once, which a larger model keeps smaller because the batch rather than the weights sets its peak memory. It sits where `image` and `audio` sit for their engines.
|
||||
- **File roles** `tokenizer` and `config` join `weights` in [`build.py`](../../surfsense_local/backend/modules/llm/catalog/local/build.py), and `validated` gains `onnxruntime` beside `llama_cpp`, `sd_cpp` and `audio_cpp`.
|
||||
- **`ModelType.EMBEDDING`.** The [classifier](../../surfsense_local/backend/modules/llm/catalog/local/classifier.py)'s embedder group, which today calls these models not runnable, gives this type instead. A GGUF embedder found through the llama.cpp search still cannot run, because the engine reads ONNX, so it stays refused there.
|
||||
- **The refresh keeps what it cannot read.** `semantic_weight` and the chunk fit come from the retrieval eval, not from Hugging Face, so [`refresh_local_manifest.py`](../../surfsense_local/backend/scripts/refresh_local_manifest.py) carries them over from the hand-authored entry, as it does every hand-authored field.
|
||||
|
||||
### A type, not a slot
|
||||
|
||||
[`model_type.py`](../../surfsense_local/backend/modules/llm/model_type.py) says a type is also a slot. `EMBEDDING` is the exception: a catalog type, so the manifest and the engines can describe it, and never a selection, so nothing can swap it. Three places enforce that, each with a test:
|
||||
|
||||
- [`selectable_for()`](../../surfsense_local/backend/modules/llm/selectable.py), which offers every type to a model nothing recognises, leaves `EMBEDDING` out;
|
||||
- `choose_model()` in [`selection.py`](../../surfsense_local/backend/modules/llm/selection.py) refuses it;
|
||||
- the `selected_models` check constraint excludes it.
|
||||
|
||||
Missing any one of them would make the embedder swappable, which is the failure this whole design exists to prevent. Only the curated list is in the manifest; a Hugging Face or remote pick never touches it, and every source still ends in the spec stored on the index row.
|
||||
|
||||
## Hugging Face
|
||||
|
||||
A live search filtered to `sentence-similarity` and `feature-extraction`. The onboarding search that exists today ([`hugging-face-search.tsx`](../../surfsense_local/frontend/src/features/onboarding/model-step/hugging-face-search.tsx)) goes through the llama.cpp engine and reads GGUF files, so this is a second search beside it, reading different files.
|
||||
|
||||
- **Runnable means ONNX.** The API process ships onnxruntime without torch ([`api.spec`](../../surfsense_local/backend/bundling/api.spec)), so a repo with only safetensors is shown as not runnable.
|
||||
- **Which file.** A generic int8 file (`model_int8.onnx`, `model_quantized.onnx`) first, then the full-precision `onnx/model.onnx` or `model.onnx`; otherwise the repo is refused. A file's external data (`.onnx_data`) downloads with it. Never a file tuned for one CPU (`*avx2*`, `*avx512*`, `*arm64*`), an `O1` to `O4` optimised variant, or an `fp16`, `q4` or `bnb4` build: fp16 runs slowly on a CPU and the rest are made for other hardware or runtimes. A bad quantisation is caught by the probe and the sanity check.
|
||||
- **Refused:** gated repos, repos Hugging Face's security scan flags, and repos without a `tokenizer.json`.
|
||||
- **Evidence:** the repo's pipeline tag and its sentence-transformers files. A repo that ships them is `declared`; one with only the tag is `inferred`.
|
||||
- **The spec is read from those files:** pooling from `1_Pooling/config.json`, normalisation from `modules.json`, prompts from `config_sentence_transformers.json`, maximum length from `sentence_bert_config.json`, width from `config.json`. Anything missing takes bge's default.
|
||||
|
|
@ -106,7 +152,7 @@ The same connections as chat and images ([ADR 0015](../adr/0015-openai-compatibl
|
|||
|
||||
For chat, images and audio, [`classifier.py`](../../surfsense_local/backend/modules/llm/catalog/remote/classifier.py) decides the type from `modalities`. That cannot find an embedder: models.dev has no embedding output, and of the 81 entries in the snapshot whose ids look like embedders, 75 declare `text → text` exactly like a chat model; the other six add image, audio or video inputs, which a text index does not use. So evidence is read in this order, and the first source that answers decides, to admit or to refuse:
|
||||
|
||||
1. **The server.** Some servers state a model's kind through their own API: LM Studio and Ollama report it per model, and Hugging Face TEI describes the one model it serves. Where the connection points at a server SurfSense recognises, a small adapter reads that answer. `declared`.
|
||||
1. **The server.** Where the connection points at a server SurfSense recognises, an adapter asks it directly. `declared`. See *Server adapters* below.
|
||||
2. **The catalog's `family`.** `text-embedding`, `cohere-embed`, `mistral-embed`, `titan-embed`, `codestral-embed`. Reliable where present, but many embedders carry `gemini`, `qwen`, `voyage` or none. `declared`.
|
||||
3. **The model id,** by the terms [`not_text_gen.py`](../../surfsense_local/backend/modules/llm/catalog/remote/not_text_gen.py) already uses to keep embedders out of chat (`embed`, `voyage`, `bge`, `e5-`, …), with an audited list of exceptions: 4 of the 81 describe themselves as chat models. `inferred`.
|
||||
4. **Nothing.** The model is not listed. The user can still name one by hand, such as a team's own server running bge. `unverified`.
|
||||
|
|
@ -115,6 +161,25 @@ The catalog supplies two more things: `context`, the longest passage the model a
|
|||
|
||||
Chat keeps refusing whatever this admits, so no model lands in both.
|
||||
|
||||
### Server adapters
|
||||
|
||||
Several servers say what a model is through an API of their own, read from their source and documentation:
|
||||
|
||||
| Server | Recognised by | Where it says | What marks an embedder | Also gives |
|
||||
|---|---|---|---|---|
|
||||
| Ollama | `GET /api/version` answers | `POST /api/show`, per model | `capabilities` holds `"embedding"`, which Ollama derives from the GGUF header's `pooling_type` | the width, as `<arch>.embedding_length` in `model_info` |
|
||||
| LM Studio | `GET /api/v1/models` answers, or `/api/v0/models` before 0.4.0 | the same list | `type` is `"embedding"` in v1, `"embeddings"` in v0 | `max_context_length` |
|
||||
| Hugging Face TEI | `GET /info` holds `model_type` | `/info`, for the one model it serves | `model_type` is `{"embedding": {"pooling": …}}`, not `classifier` or `reranker` | `model_sha`, the exact revision; `max_input_length`; `POST /tokenize` |
|
||||
| OpenRouter | its host | `GET /api/v1/embeddings/models` | `output_modalities` is `["embeddings"]` | `expiration_date`, `hugging_face_id`, `context_length` |
|
||||
|
||||
Three things follow from how these servers behave:
|
||||
|
||||
- **An adapter can list, not only classify.** Discovery today reads `GET {base_url}/models` ([ADR 0015](../adr/0015-openai-compatible-connections.md)). TEI has no such route, so its connection would find nothing, and OpenRouter's `/models` leaves every embedder out. For both, the adapter's own list is the listing.
|
||||
- **Adapters reach past `/v1`.** A connection's base URL ends in `/v1`; Ollama's `/api/*`, LM Studio's `/api/*` and TEI's `/info` sit at the server's root, so the adapter strips it. Same host, so no consent beyond the connection's own.
|
||||
- **Older versions are expected.** Ollama has reported `capabilities` on `/api/show` since April 2025 but on `/api/tags` reliably only since 0.34.1 in September 2026, so the adapter asks `/api/show`. LM Studio's v1 API arrived in 0.4.0; the adapter falls back to v0 and accepts both spellings.
|
||||
|
||||
llama-server and vLLM say nothing: llama-server's `/props` and `/v1/models` carry no kind, its router mode reports every model's output as `text`, and vLLM's `/v1/models` has no task. Both, and Ollama, also return vectors from a chat model: Ollama's embed route asks for no capability, llama-server embeds with any model once started with `--embeddings`, and vLLM can run a chat model as an embedder. They get no adapter, and the catalog, the name, the probe and the sanity check decide. LiteLLM's `/model/info` (`mode` is `"embedding"`) and Infinity's `/models` (`capabilities` holds `"embed"`) can join the table later.
|
||||
|
||||
### Calling it
|
||||
|
||||
- **Provider notes.** models.dev does not record a provider's query and document switch or its batch limit. A short table in `modules/embedding/` keyed by provider holds them; a provider not in it gets plain `/embeddings`.
|
||||
|
|
@ -153,7 +218,7 @@ Migration `0001` creates `chunk_vectors` at `SURFSENSE_LOCAL_EMBEDDING_DIMENSION
|
|||
|
||||
[`chunking.py`](../../surfsense_local/backend/worker/ingestion/chunking.py) measures chunks with the embedder's tokenizer. If the embedder decides the boundaries, changing it re-cuts every document, chunk ids change, and every citation in every old chat breaks. The chunker gets its own pinned tokenizer, bge's file under its own name, which changes nothing today.
|
||||
|
||||
A curated model is admitted only if chunks fit it. A Hugging Face pick is not checked in advance, so ingest counts, with the model's own tokenizer, the passages longer than `max_tokens`; they are embedded cut short, and Settings shows the count. A remote model's tokenizer is not on the machine, so an overlong passage shows up only as the provider's error.
|
||||
A curated model is admitted only if chunks fit it. A Hugging Face pick is not checked in advance, so ingest counts, with the model's own tokenizer, the passages longer than `max_tokens`; they are embedded cut short, and Settings shows the count. A remote model's tokenizer is not on the machine, so an overlong passage shows up only as the provider's error, unless the server counts tokens itself, as TEI's `/tokenize` does.
|
||||
|
||||
### One write path, one read path
|
||||
|
||||
|
|
@ -186,16 +251,16 @@ Checked against the active row, not a flag on the file, so a model retired by a
|
|||
| Hugging Face | repo deleted or made private since | no repair until [changing](embedding-model-change.md) ships |
|
||||
| remote | offline, outage, key revoked, host turned off | chat answers `503` saying why, ingest fails with Retry |
|
||||
| remote | provider retires the model | no repair until [changing](embedding-model-change.md) ships |
|
||||
| remote | a models.dev refresh marks it `deprecated` | Settings warns before the provider turns it off; models.dev rarely sets it, so this is a bonus, not a guarantee |
|
||||
| remote | OpenRouter gives it an `expiration_date`, or a models.dev refresh marks it `deprecated` | Settings warns before the provider turns it off; OpenRouter's date is real, models.dev's status is rarely set, so neither is a guarantee |
|
||||
|
||||
bge is bundled and cannot become unreachable. Chat's check for missing files in [`chat/router.py`](../../surfsense_local/backend/modules/chat/router.py) reads the active spec, not bge's file list.
|
||||
|
||||
## Onboarding
|
||||
|
||||
1. A new step before the chat model. It is not a model slot: [`slot.ts`](../../surfsense_local/frontend/src/features/onboarding/model-step/slot.ts) is typed to `ModelType`, and the embedder is not a selection. The step list needs a kind of step that is not one.
|
||||
2. Curated models come first, bge preselected, each with its size and one line on what it is for: *"Choose Multilingual only if your documents or questions are in more than one language."* Hugging Face search and remote models sit below.
|
||||
2. Curated models come first, bge preselected, each with its download size and one line on what it is for: *"Choose Multilingual only if your documents or questions are in more than one language."* Hugging Face search and remote models sit below.
|
||||
3. A local pick not on disk resolves first (size, whether it is on disk, any error), then downloads with the hash check. A remote pick connects. Then the probe and the sanity check run, and the pick gets its label. The click is a user action, so the egress dialog may appear once.
|
||||
4. Before the user continues, the step says what cannot be undone: that the model is fixed for this library, what its label means, and for remote, where every document goes.
|
||||
4. Before the user continues, the step says what cannot be undone: that the model is fixed for this library, what its label means, and for remote, where every document goes. A Hugging Face pick shows its download size with one line: *"Larger models make adding documents slower and use more memory, and this choice can't be changed later."*
|
||||
5. Finishing onboarding writes the `embedding_indexes` row and creates the vector table. Nothing chosen, or a pick whose download or checks did not finish, writes bge.
|
||||
|
||||
## Settings
|
||||
|
|
@ -210,9 +275,9 @@ Onboarding is already complete, so they never see the step. A migration writes a
|
|||
|
||||
| | Curated (granite) | Hugging Face | Remote |
|
||||
|---|---|---|---|
|
||||
| Disk | about 118 MB on top of bge's 63 MB | whatever the repo weighs | none |
|
||||
| Disk | 98 MB for the 97M model, 313 MB for the 311M, on top of bge's 63 MB | whatever the repo weighs | none |
|
||||
| Memory | a copy per embedding process | the same | none |
|
||||
| Ingest speed | slower than bge; granite has 3× the parameters and is not yet measured | unknown | the network and the provider's rate limit |
|
||||
| Ingest speed | depends on the build: an int8 build can run faster than the bundled FP16 bge, a larger model runs slower | depends on the model and its build | the network and the provider's rate limit |
|
||||
| Ranking | measured weight | 0.65, not measured | 0.65, not measured |
|
||||
| Privacy | stays on the machine | stays on the machine | every passage and question leaves it |
|
||||
| Money | none | none | billed per passage indexed, retries included |
|
||||
|
|
@ -222,12 +287,14 @@ A model with another width also changes storage: 1536 dimensions is 6 KB a vecto
|
|||
|
||||
## Rejected
|
||||
|
||||
- **The embedder as a `selected_models` slot.** Those rows are preferences, changed freely through `choose_model()` and the model-slot Settings UI, and cascaded away with their connection. The embedder is a property of the index; putting it there offers exactly the swap that breaks search.
|
||||
- **The embedder as a `selected_models` slot.** Those rows are preferences, changed freely through `choose_model()` and the model-slot Settings UI, and cascaded away with their connection. The embedder is a property of the index; putting it there offers exactly the swap that breaks search. It is a model type in the catalog, and nothing more.
|
||||
- **A separate manifest for embedders.** It would keep `ModelType` purely a slot, at the price of a second refresh script, a second pinning and licence pipeline, and a second install, download and delete path, all duplicating what the local catalog already does. Three guarded places are cheaper than a second catalog.
|
||||
- **Admitting an unknown model, as the other types do.** Their rule is that the user sees the model answer before trusting it. An embedder never answers where the user can see it, so the sanity check stands in for that look.
|
||||
- **Trusting the name alone, or the probe alone.** The name terms let chat models through, and a probe succeeds on a chat model served as an embedder.
|
||||
- **Change freely, repair by re-upload.** [Unsloth Studio](https://github.com/unslothai/unsloth/tree/main/studio) does this: any Hugging Face embedder at any time, identity stamped on each document, one vector table dropped outright on a width change, and *"Re-upload existing ones after changing the model."* After a switch, older documents drop out of meaning search without saying so. It suits a studio where RAG is a side feature; here the library is the product. Taken from it: the identity stamp on each document, a resolve step before saving, and pooling as part of the identity.
|
||||
- **Locking at the first indexed chunk** rather than at onboarding. It lets Settings change the model while the library is empty, but most users add a source within minutes, so the window is short and the rule is harder to explain. One moment, onboarding, is simpler.
|
||||
- **Curated local models only.** Safer, since every choice would be measured and none could vanish, but it leaves out users with a hosted embedder they already pay for and multilingual models not yet on the list. The risk is taken knowingly and stated in onboarding.
|
||||
- **A size or memory cap.** It would need measuring every model on some reference machine, and would still say little about the user's own. The curated list is ours to keep light, and a Hugging Face pick shows its size and says what a larger model costs.
|
||||
- **Falling back to bge when the locked model is unreachable.** It would keep search answering, with wrong answers.
|
||||
|
||||
## When this ships
|
||||
|
|
@ -235,15 +302,15 @@ A model with another width also changes storage: 1536 dimensions is 6 KB a vecto
|
|||
- A new ADR amending [ADR 0007](../adr/0007-bundled-embeddings.md): the index records its model, the width comes from the row, bge is the default rather than the only model, and remote embedding is no longer a later opt-in.
|
||||
- [ADR 0015](../adr/0015-openai-compatible-connections.md) says embeddings do not go through connections; the same ADR amends that.
|
||||
- [search](../architecture/search.md), [documents](../architecture/documents.md), [data model](../architecture/data-model.md) and [egress](../architecture/egress.md) describe the index row, the spec, the lock and the background consent.
|
||||
- The [local catalog](../architecture/local-models/catalog.md) describes the fourth engine, the `embedding` entry block, and `EMBEDDING` as a type that is not a slot.
|
||||
- [retrieval](retrieval.md) proposes granite as the bundled replacement for bge. This proposal keeps bge as the default and offers granite as a choice; that proposal's measurements carry over, and its migration section is superseded by [changing](embedding-model-change.md).
|
||||
|
||||
## Open questions
|
||||
|
||||
- The sanity check's threshold and its fixed strings. Measured, not guessed: run it on known embedders (bge, granite, OpenAI, Cohere) and on chat models served as embedders, and set the bar between them. Whether English strings are enough for a multilingual model.
|
||||
- Which server adapters ship first, and the exact field each reads; LM Studio's, Ollama's and TEI's are to be pinned against their current versions.
|
||||
- Granite's `max_tokens`, and how many of its tokens a 480-token bge chunk becomes in Hindi, Japanese and Chinese.
|
||||
- What the adapters could not be checked for without a running server: whether LM Studio's `/v1/embeddings` refuses an `llm` model, whether its v1 `key` equals the id its `/v1/models` returns, and Infinity's default URL prefix.
|
||||
- How many tokens a 480-token bge chunk becomes in each curated model's tokenizer, in Hindi, Japanese and Chinese. Every candidate takes 32,768, so the question is the admission check, not a likely failure.
|
||||
- Whether granite is ready: [retrieval](retrieval.md) asks for the same-language slices to be hardened first.
|
||||
- The size cap for a curated entry, and whether a Hugging Face pick above it is refused or warned, given three processes each load it.
|
||||
- Which ONNX file a Hugging Face repo with several is read from, and whether a quantised one is ever preferred.
|
||||
- The remote context floor: whether a model whose models.dev `context` is close to `CHUNK_TOKENS`, such as Cohere v3 at 512, is refused, since another tokenizer can count a chunk as more tokens.
|
||||
- The smallest way to make a bad pick recoverable before full [changing](embedding-model-change.md): a blocking rebuild that pauses search and re-embeds every chunk into a new index row.
|
||||
- How close a pinned build's vectors must be to the original model's to count as a match.
|
||||
- Harrier's licence lineage: harrier-270m is a Gemma 3 architecture under an MIT card, so whether Gemma's terms reach it needs checking before it is listed.
|
||||
- The remote context floor: whether a model whose models.dev `context` is close to `CHUNK_TOKENS`, such as Cohere v3 at 512, is refused, since another tokenizer can count a chunk as more tokens.
|
||||
Loading…
Add table
Add a link
Reference in a new issue