Commit graph

6 commits

Author SHA1 Message Date
ybai08
14627e28d8 feat(local): give an embedder repo's refusal a code the interface can translate
An opened Hugging Face embedder repo that cannot run says why in a sentence
the backend words, so it was English in every language. Four of its five
refusals now carry a NotAnEmbedderCode in the row's not_runnable_code:
gated, no ONNX build, no tokenizer.json, flagged by Hugging Face's scan.
The interface has a line for each, in ten languages. The fifth names a
file and keeps travelling as its English sentence.

Refs #1967
2026-10-08 16:59:57 -05:00
ybai08
87b8ce48dc docs(local): keep the FP16 note in ADR 0007's Status line and fix the cosine's scope
An accepted ADR takes only Status-line and link edits, so the Correction
section goes and its fact moves to the Status line. ADR 0036 linked to the
Known gaps line this branch deletes; it now links to where the note lives.
The cosine against the int8 export was a mean over passages paired with a
minimum over queries: state both figures over the 66 queries, and the
passages' own beside them.
2026-10-04 19:05:48 -05:00
ybai08
21545575f2 docs(local): scope the re-embed sentence to libraries that use bge-small 2026-10-02 17:05:13 -05:00
ybai08
a3ab727f26 docs(local): say the bundled embedder is FP16, with its measured size and speed 2026-10-02 16:40:36 -05:00
Anish Sarkar
4259345fe3 refactor(local): split the onboarding model step into folders
Move the model step's files into kinds/ (slot, step kind, step models, and the per-kind hooks), local/ (Hugging Face search, local choices and list) and server/ (server option and path). Move the per-step copy out of model-step.tsx into step-copy.ts. Rename use-slot-delete.ts to step-deletes.ts, since it also handles deletes on the embedding step.

Reword the embedding copy to say plainly that the choice is permanent:
- the step body says what an embedding model is for
- the notice says it can't be changed later, and that the default works well for English
- the search note no longer repeats that the choice is permanent
- Settings says the model was chosen in onboarding and can't be changed

Update all ten translations, the tests and the doc links to match.
2026-10-02 01:57:57 +05:30
Anish Sarkar
84c7ffad18 feat(local): choose the embedding model in onboarding
The embedding model is now chosen in the last onboarding step and fixed
for the whole library. Skipping keeps bge-small, which ships with the
app and can't be deleted. Changing the model later is not built.

- An embedding_indexes row records the model's spec and its vector
  table, and each document records which index embedded it. Migration
  0021 stamps existing libraries with bge-small. Routes that queue
  embedding work return 409 embedding_not_chosen until a model is
  chosen.
- Ingest, Studio and search embed through the active index; the old
  worker embedding module is removed. Workers wait for the schema, and
  documents interrupted when the app closed are marked failed.
- Curated embedders (bge-small, granite-97m, granite-311m) live in the
  local manifest, run by a new onnxruntime engine. EMBEDDING is a
  catalog type, never a selection slot.
- Hugging Face search finds ONNX embedders. Each download is checked: a
  probe sets its width, and a ten-question search check must pass.
  Small tokenizer.json files that Hugging Face lists without a checksum
  are hashed at the pinned commit. Results use the GGUF search's shapes.
- Onboarding renders the embedding step with the same component and
  shared list as the other steps; only its choice hook differs. Use
  marks the choice and Finish sends it. Bundled models have no Delete.
- Settings has a read-only Embedding section.
- ADRs 0036 and 0037, a new embedding architecture doc, and updates to
  the docs it touches. New strings are translated into every language.
2026-10-02 01:45:10 +05:30