SurfSense/docs/architecture/model-capabilities.md
DESKTOP-RTLN3BA\$punk 4dd3ce2f66 docs: cover the work since 2.1.0 so others can pick it up
- chat.md, overview.md: how a streamed token renders only the live reply
  (runSummaries, the 32 ms notice gap, LiveThreadRuntime, the bounded
  highlighter, memoized panels) and that the columns resize.
- model-capabilities.md, agent.md, data-model.md, 07-dev-setup.md: the two
  chat modes, their gates and defaults, where a pick is stored, the 0-id fix,
  and how to re-run the OpenRouter sweep and regenerate capabilities.json.
- RELEASE.md: the steps 2.2.0 was released by, from the bump (with its
  PowerShell form) to the tag on the merge, the Known issues, an optional
  update check and publishing with --latest; dry runs are signed.
- 06-repo-readme.md, AGENTS.md, README.md's comment: the README's structure,
  how to change it and its translations, re-checking the format table, and
  replacing the demo video.
- Stale statements fixed (the Apple agreement, opencode never shipped, three
  languages, Studio's exec(), proposal statuses, the roadmap), and each gap
  still open after the release in its doc's Known gaps, linked to #2219-#2230.
2026-10-11 17:55:51 -07:00

17 KiB
Raw Permalink Blame History

Model capabilities

What SurfSense measured of a chat model, never its size or its prompt tier, decides two things: the mode a new chat starts in, and Studio's Word and PDF path. A list shipped with the app gives each measured model one of four levels; one function reads it for the selected model and its connection. No level takes a mode away: every model may choose Agentic, and only a technical gate keeps one out (modes). An unmeasured model is labelled "Not tested".

Code: surfsense_local/backend/modules/llm/capability/, surfsense_local/backend/scripts/capability_list/, surfsense_local/frontend/src/features/models/capability/, surfsense_local/frontend/src/features/chat/modes/

The levels

Level Means New chats Studio's Word and PDF
agent measured, and passed the bar start Agentic a script
agent_limited measured, near the bar start Agentic, labelled "may need nudges" a script
studio_only measured, and failed start Basic (Q&A), Agentic offered with the score Markdown
not_measured no row holds for it start Basic (Q&A), Agentic offered as untested, or as a local copy of a model that passed Markdown served from this computer, a script from a remote host

The bar, in verdict.py: the smoke and the multi-turn demo pass, at least 80% of the counted cases pass, and no failure was "made no document" or "looped". Near the bar is the same with at least 60%. Anything else measured is studio_only. A case that needs image input is not counted against a text-only model, and a case a gate stopped counts as not passed.

The list

measured/capabilities.json, schema version 1 (schema.py), is read once per process and never fetched; a list that fails to load makes every model not_measured. Each row holds the canonical key, its matching rules (match.keys, and match.served: where the model must be served for the row to hold), the level, the suite and its version, the date, the provider and host the runs went through, the id they sent, reads_images, the pass counts and a one-line English note. The whole list carries provisional.

Today's list has 270 rows. The 11 models of the first ladder results (08), suite create-and-edit v1, screening runs of one per cell: Sonnet 5.5, Opus 5.5, Kimi K3, GLM-5.3, DeepSeek V4 Pro and Qwen3.8-27B are agent; Haiku 4.5 is agent_limited; Qwen3.6-35B-A3B, Gemma 4 31B, Ministral 14B and Qwen3.5-9B are studio_only. Then 208 rows from the OpenRouter screening sweep of 7 and 8 Oct 2026 (tests/live/sweep), suite openrouter-screen v1: every OpenRouter model that calls tools, holds 32,768 tokens and costs under $3/M of input, run on the smoke case, then the PDF brief and the board pack, one run each with one re-run of a split; 106 passed both and are agent, 102 are studio_only. The 16 OpenAI models an id of 0 had failed, and two Grok models that had passed only on a re-run, were run again once the tools took an optional id of 0 as left out (agent). The sweep cost $65.41, the re-run $4.68 of it. And 51 assumed rows, never run: models from $3/M of input, the reasoning flagships whose output costs $10/M or more, and the models whose case went past the sweep's $3 stop. 11 models the providers kept rate-limiting have no row and are untested.

Two more suites add rows after the ladder's. openrouter-screen v1 is a screening on OpenRouter: a smoke check, then two cases, PDF brief and Board pack. Its row is agent when both pass and studio_only otherwise, with the counts (passed of counted: 2) and a note written from them; a pass is served: remote, a failure both. assumed lists a flagship the sweep did not run, at agent: one at $3 or more per million input tokens, one named with the sweep's --assume, or one whose case reached the per-case dollar cap (07-dev-setup): passes 0 of 0, note "Not run: an expensive flagship assumed to pass", and reason assumed rather than counts. The schema refuses an assumed row with runs or below agent.

The list is written, never edited: python scripts/write_capability_list.py [results.json] (from surfsense_local/backend, with . on the path) applies the bar to a ladder results file and rewrites it. The input's shape is in ladder_input.py: per model, the id sent, provider, host, where it was served, date, reads_images, a note, and one outcome per case (pass, fail, fail:<behaviour> from 08's grouping of failures, n/a, not_run). The committed input is capability_list/ladder/create-and-edit-v1.json, and a test fails when the shipped list's ladder rows are not what the generator writes from it. Rewriting them keeps every other suite's rows that no ladder row names.

python scripts/catalog_rows.py sweep-results.json [--date YYYY-MM-DD] (same place) adds the sweep's rows: each screened model, then each assumed one. The input's shape is in sweep_input.py: models, each with id, key, reads_images, status, level, smoke, pdf_brief, board_pack, passed, counted, run, measured_on, cost and notes; unfinished; and assumed, each with id, key and reads_images. Only a model the sweep has a verdict for (status done, or no status and every count) gets a row, so a case left unresolved by a provider's errors never reads as a failure; the rest are printed as left out. Its level must agree with its counts. A row's date is its own measured_on, else the top-level date, else --date, else today. The list keeps one row per model, by its key or by the looser one, and the stronger row stays, listed or added: the ladder's 8-case rows stand over a screening, and a screening over an assumption, so a later sweep can measure a flagship it assumed. Between rows of one suite the shorter key, the model's own name, wins (gpt-5-2 over gpt-5-2-chat), whichever run came first, and a listed row stays over an added one with the same key; the script prints what each row replaced or why it was left out. A variant whose own model the sweep has no verdict for yet waits for it, rather than stand for it. A results file written before the sweep took a capped model as assumed (7 Oct 2026) counted a capped case as failed, and such a row's note says how many were. The sweep's own notes and cost stay in its file. How to run the sweep and then this script, and how to take a re-run's verdict for a model the list already screened, is in 07-dev-setup.

Matching

capability_of(model, connection) in resolve.py is synchronous and offline. It reduces the id to a canonical key (model_key.py): lower case, the provider prefix and a @ version dropped, OpenRouter's routing suffixes (:free, :nitro, :floor) dropped, a dot between digits made a hyphen, and an eight-digit snapshot date dropped. So claude-haiku-4-5 and claude-haiku-4-5-20251001 on Anthropic and anthropic/claude-haiku-4.5 on OpenRouter are one model. An id starting with ~ or ending in latest is an alias and never measured (reason alias). Any other suffix, such as :thinking, keeps its own key.

When no row has that key, a looser one, used only to find a row, matches how another server spells the model (match_key.py). It also drops a file path's folders, a file extension (.gguf), quantisation tags (Q4_K_M, UD-Q4_K_XL, IQ4_XS, fp8, bf16, int4, awq, gptq, mlx, gguf, exl2, qat, 4bit and the like) and the instruct, it and chat tags, reads Ollama's name:tag as name-tag, and puts a hyphen between a letter and a digit. So qwen3.8:27b, Qwen/Qwen3.8-27B-Instruct, Qwen3.8-27B-Q4_K_M.gguf and unsloth/Qwen3.8-27B-GGUF find the qwen3-8-27b row, and gemma4:31b finds gemma-4-31b-it. Every size and version stays: qwen3.8-14b, qwen3.5-27b and qwen3-8b find nothing. The list refuses two rows that one looser key would find.

A pass holds only where it was measured. Every row today was measured on a remote host, so a model served from this computer (llama.cpp, or a loopback connection such as Ollama or LM Studio) or from a server on the user's own network (a private, link-local or carrier-grade NAT address, a single-label name, or a .local, .lan, .internal or .home.arpa name) with a passing row is not_measured, reason measured_elsewhere: a quantized copy there is a measurement not yet made. Such a server counts as local for match.served only; Studio's rule for a model not measured still counts it remote. A failure holds everywhere, since a copy of one's own is the same model or a smaller one.

The result carries the level, a Reason(code, values) (measured_pass, measured_near or measured_fail with the pass counts, suite version and date; assumed, with none; measured_elsewhere with the host; alias; no_row) and the row.

What follows from it

  • A new chat's default mode (modes): Agentic for agent and agent_limited, Basic (Q&A) for studio_only and not_measured.
  • Studio (strength.py): writes_script() is true for agent and agent_limited, false for studio_only, and the local or remote rule for not_measured.

The modes

A chat is Basic (Q&A), the chat engine's retrieval answers with Studio beside it, or Agentic, the agent. It is chosen when the chat starts and kept (which threads get it). new_chat_modes(selected, facts) (modes/new_chat.py) says, synchronously and offline, what a new chat on the selected model may be:

  • agentic_allowed, and when not, blocked: a technical gate only, never a score. agent_not_installed when no opencode is staged; tool_calls_unsupported when the remote catalog states tool_call: false, or llama-server does not report supports_tool_calls for a local model's template; window_below_floor when the stated window is under 32,768 tokens (agent_gate.py). A catalog that says nothing, and a runtime that cannot be read, allow it. A local model's tool calls and window are read when a chat starts, since reading them would load the model, so the selection read allows it until then; a chat refused by one of these gates disables Agentic for that model in the composer until the app restarts.
  • default_mode: the mode the user last chose for a new chat on this model; else Agentic for agent and agent_limited, which hold only on a remote host; else Basic. A blocked Agentic defaults to Basic.
  • reason, what the switch says beside Agentic: measured_pass or measured_near with passed and counted; measured_below, a low scorer, with the same; assumed; local_copy, a copy on the user's own machine or network of a model whose row passed on a remote host, with that host; untested. An assumed row says assumed on a local copy too, since it was run nowhere, and the capability's own reason is assumed there rather than measured_elsewhere.
  • remembered_mode, under chat_modes in the text selection's settings, one entry per provider, connection and model name (modes/remembered.py), written when a chat is opened with a mode the user chose (remember: true). A default sent back unchosen is not written, so a later score or the developer switch turned off still decides. The entries stay when the slot takes another model, so a model's choice comes back with it. The agent_trial the modes replaced reads as its model's Agentic, kept as such when the slot changes, and the first write drops it.

SURFSENSE_LOCAL_AGENT_UNTESTED_MODELS=1 makes Agentic the default for every model the user has not picked a mode for and lifts the window floor; a stated no on tool calls, or no opencode, still keeps Agentic out.

API

  • SelectionRead.capability on GET and PUT /llm/selection/text_gen (null for other slots): level, label_key (the ICU select branch), reason (code, values), note, measured (key, suite, assumed, suite_version, measured_on, provider, host, reads_images, passed, counted, provisional, or null) and modes (agentic_allowed, blocked, default_mode: basic or agentic, reason (code, values), remembered_mode).
  • capability_level on each row of GET /llm/connections/{id}/models and GET /llm/providers/llamacpp/models that can fill the chat slot.
  • POST /workspaces/{id}/chat/threads takes mode and remember (chat).

In dev builds only, the model picker tags each chat model by how it did in the Agentic tests: "Agentic", "Agentic, may need nudges", "Low Agentic score" or "Not tested"; production shows no tag. Settings › Models shows the chat model's level, its evidence line and its note, or for an assumed row only "Not run: SurfSense expects this expensive flagship model to pass."; then the mode new chats start in, and either that the mode is chosen per chat in the composer or why Agentic is not available.

The composer's switch (mode-picker.tsx) is a menu of "Basic (Q&A)" and "Agentic", each with one line on what it does. On a new chat it starts at the model's default_mode and follows the model when it changes; a pick holds for that model for the session, in a context above the open workspace. The new thread's mode is always sent, with remember: true only when the user picked it, so only a pick becomes the model's default. An Agentic that did not pass carries a tag ("Not tested", "Low score", "May need nudges", "Local copy", "Expected to pass"); the reason, or a gate, shows in a tooltip beside the Agentic row on hover or focus and is read with the row, in both menus: "Passed 2 of 8 Agentic tests, so it may stop early or make mistakes.", "Not tested with this model yet, so it may stop early or make mistakes.", "Passed on its full-size version. A local copy may do worse.", or that a flagship is assumed. Nothing is said under the composer. A gated Agentic is disabled with its reason, such as "This model can’t use tools." In an open chat, under "This chat’s mode", the switch lists the same two rows with the chat's mode checked; the other is marked "New chat", and picking it picks that mode and opens a new chat. A refused Agentic chat is worded by its code in a toast, and the question and images go back into the composer. A refusal by a gate the selection could not say, such as a local model's tool calls, disables Agentic with its reason for that model for the session (new-chat-mode.ts), so the next chat starts in Basic (Q&A) rather than asking again. The agent reads no attached image yet, so an Agentic chat, new or open, takes none (chat).

Known gaps

  • Every row is screening (n=1) on OpenRouter's default routing or Anthropic's compatibility layer; 05's committed rows, three runs per cell with the provider order pinned, replace it.
  • No row is measured on this computer, so every local model is not_measured; llama.cpp runs need a served: local row.
  • The matrix runner writes per-run result.json files, not the ladder input; each column is mapped by hand, triage labels included.
  • The sweep's rows come from a committed input, capability_list/sweep/openrouter-screen-v1.json: the screen of 7 and 8 Oct 2026 with the 18 re-run models' verdicts in place of their first ones. Its two runs are beside it under sweep/runs/, each a sweep-results.json and its attempts.jsonl; their run folders, about 840 MB of transcripts and files, stay out of the repository. catalog_rows.py cannot fold a re-run into a screen, so that input was put together by hand, and no test pins the sweep's rows to it as one pins the ladder's (#2221). The 11 models the sweep left unresolved have no row (#2222).
  • The row's note is English; the interface shows it untranslated.