mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-10-19 15:44:27 +00:00
docs: cover the work since 2.1.0 so others can pick it up
- chat.md, overview.md: how a streamed token renders only the live reply (runSummaries, the 32 ms notice gap, LiveThreadRuntime, the bounded highlighter, memoized panels) and that the columns resize. - model-capabilities.md, agent.md, data-model.md, 07-dev-setup.md: the two chat modes, their gates and defaults, where a pick is stored, the 0-id fix, and how to re-run the OpenRouter sweep and regenerate capabilities.json. - RELEASE.md: the steps 2.2.0 was released by, from the bump (with its PowerShell form) to the tag on the merge, the Known issues, an optional update check and publishing with --latest; dry runs are signed. - 06-repo-readme.md, AGENTS.md, README.md's comment: the README's structure, how to change it and its translations, re-checking the format table, and replacing the demo video. - Stale statements fixed (the Apple agreement, opencode never shipped, three languages, Studio's exec(), proposal statuses, the roadmap), and each gap still open after the release in its doc's Known gaps, linked to #2219-#2230.
This commit is contained in:
parent
f6a3c97272
commit
4dd3ce2f66
40 changed files with 358 additions and 150 deletions
|
|
@ -141,7 +141,7 @@ its lock entry is valid.
|
|||
|
||||
- Do not carve hosted code out of backend/web until the purge on 18 Oct 2026. The export window and the purge script run on that code until then.
|
||||
- Do not drop SearxNG, sandbox, OpenSandbox, or zero-cache from compose until a new compose is defined.
|
||||
- Edit the root `README.md` directly. The draft at `plans/community-local/seo/drafts/README.md` shipped on 18 Sep 2026 and is kept only for history. Anything added to the README replaces something — its length was measured against the eight biggest repos in this category.
|
||||
- Edit the root `README.md` directly. The draft at `plans/community-local/seo/drafts/README.md` shipped on 18 Sep 2026 and is kept only for history. Anything added to the README replaces something — its length was measured against the eight biggest repos in this category. Make the same change in every `README.<locale>.md`, section for section; [`06-repo-readme.md`](plans/community-local/seo/06-repo-readme.md#the-readme-since-9-oct-2026) has the rules, where each format-table cell is checked, and how to replace the demo video.
|
||||
- Do not commit, push or open a PR unless asked.
|
||||
|
||||
## Security
|
||||
|
|
|
|||
13
README.md
13
README.md
|
|
@ -1,11 +1,14 @@
|
|||
<!--
|
||||
Rationale: plans/community-local/seo/06-repo-readme.md. Anything added replaces
|
||||
something (AGENTS.md). Ship the agent copy only with the first release whose
|
||||
installer carries the agent; v2.1.0 does not.
|
||||
The format table is checked against origin/dev c5ff7938a; re-check each cell before a release.
|
||||
Rationale and how to change it: plans/community-local/seo/06-repo-readme.md. Anything
|
||||
added replaces something (AGENTS.md). The agent copy holds from v2.2.0, the first
|
||||
release whose installers carry the agent.
|
||||
The format table is checked against origin/dev c5ff7938a, which v2.2.0 contains; re-check
|
||||
each cell against the release commit before a release.
|
||||
The demo video under the header is a GitHub upload, not a file in the repo. To replace it, drag an
|
||||
MP4 under 10 MB into any comment box in this repo, copy the user-attachments link and swap the URL.
|
||||
Every README.<locale>.md follows this file section for section; change them together.
|
||||
Every README.<locale>.md follows this file section for section and names the app's UI in
|
||||
the words of surfsense_local/frontend/translations/<locale>.json (Arabic: the English in
|
||||
parentheses); change them together.
|
||||
Time-boxed: delete the sunset callout on 18 October 2026, here and in every README.<locale>.md.
|
||||
-->
|
||||
|
||||
|
|
|
|||
|
|
@ -4,20 +4,21 @@ What the maintainers are working on, grouped by when. This page names the initia
|
|||
|
||||
## Now
|
||||
|
||||
- **Release and CI health.** v2.0.3 was the first release to carry the llama.cpp, sd.cpp and audio.cpp runtimes, built and packaged on all three runners, and [`desktop-tests.yml`](../.github/workflows/desktop-tests.yml) runs the desktop backend, frontend and Electron tests on pull requests. What is left is the CI that proves the runtimes: the release workflow still accepts a prerelease version. See [packaging](architecture/packaging.md) and [updates](architecture/updates.md).
|
||||
- **Release and CI health.** v2.0.3 was the first release to carry the llama.cpp, sd.cpp and audio.cpp runtimes, built and packaged on all three runners, and v2.2.0 the first whose installers carry opencode, signed on Windows and notarized on macOS. What is left is CI that catches a break before a release, and the installers' third-party notices ([about](architecture/about.md)). See [packaging](architecture/packaging.md#known-gaps) and [updates](architecture/updates.md).
|
||||
- **Egress gaps.** The follow-up fetch for a remote image URL leaves without a consent decision, and some stored grants no longer match what Settings › Network shows. See [egress](architecture/egress.md).
|
||||
- **Import from cloud.** The hosted export window closes on 18 Oct 2026. Imports need a progress summary, and a re-run must bring back the chat threads an interrupted import missed. See [import](architecture/import.md).
|
||||
- **Scraper API and MCP license mode.** PATs are purged on 18 Oct 2026; after that, license mode is how scraper API and MCP users keep access. See [contract 2](contracts/02-scraper-api-auth.md).
|
||||
- **Model catalog.** One catalog for local and remote models, each classified offline from a reviewed, packaged manifest and keyed by model type. See the [proposal](proposals/model-catalog.md).
|
||||
- **Hosted wind-down to 18 Oct 2026.** `/sunset` lacks the deletion date, the refund offer and the MCP change; background jobs keep running through the export-only tail; and the purge is still ahead. See [sunset](architecture/sunset.md), the [license portal](architecture/license/portal.md) and the page plan in [`plans/community-local/portal/02-pages.md`](../plans/community-local/portal/02-pages.md).
|
||||
- **Plugins.** Redesigned: a plugin is an MCP server the agent, the chat engine and the user call through one Tool Gateway, remote servers hosted by their publishers first, with SurfSense's paid scrapers as the first paid plugin once the scraper API has license mode. The earlier local runner is kept for bundles later. See the [proposal](proposals/plugins/README.md) and ADRs [0051](adr/0051-plugins-are-mcp-servers-behind-one-tool-gateway.md) to [0053](adr/0053-plugins-are-listed-in-a-registry-and-third-party-ones-start-off.md).
|
||||
- **Agent.** In progress: a chat thread opened in Agentic mode is the agent's, answered in steps by opencode over the workspace's extracted text, with SurfSense's search and Studio, on any model a technical gate, such as a stated no on tool calls, does not keep out. opencode ships in every installer and dev run, and models are measured against the agent's bar ([model capabilities](architecture/model-capabilities.md)); a model below it will also run fixed workflows with structured output. Outputs first: nothing edits the user's files. See [agent](architecture/agent.md), the [proposal](proposals/agent/README.md) and [ADR 0028](adr/0028-model-written-code-runs-with-approval.md).
|
||||
- **Plugins.** Redesigned: a plugin is an MCP server the agent, the chat engine and the user call through one Tool Gateway, remote servers hosted by their publishers first, with SurfSense's paid scrapers as the first paid plugin once the scraper API has license mode. The earlier local runner is kept for bundles later. See the [proposal](proposals/plugins/README.md) and ADRs [0051](adr/0051-plugins-are-mcp-servers-behind-one-tool-gateway.md) to [0054](adr/0054-surfsenses-plugins-run-on-one-plugin-host.md).
|
||||
- **Agent.** In progress: a chat opened in Agentic mode is the agent's, answered in steps by opencode over the workspace's extracted text, on any model a technical gate, such as a stated no on tool calls, does not keep out. It makes Word, PDF, PowerPoint and Excel files as scripts SurfSense runs, revises the user's own Word, Excel and PowerPoint files into new copies, Word's as tracked changes, works on PDFs and analyses data; the user's file itself is never written. opencode ships in every installer and dev run, and a list of measured or assumed models sets each chat's default mode ([model capabilities](architecture/model-capabilities.md)); a model below the bar will also run fixed workflows with structured output. The [file agent proposal](proposals/file-agent/README.md) orders what comes next: engines, versions, jobs such as the contract redline, and linked folders. See [agent](architecture/agent.md), the [agent proposal](proposals/agent/README.md) and [ADR 0039](adr/0039-document-scripts-run-without-approval.md).
|
||||
- **Docker.** The desktop stack without Electron as one container, replacing the legacy self-host compose. The CORS origins setting, the `VITE_API_BASE` fallback and boot outside `index.ts` are in; the headless entry is next. See the [proposal](proposals/docker/README.md) and [ADR 0035](adr/0035-docker-compose-runs-the-desktop-stack.md).
|
||||
|
||||
## Next
|
||||
|
||||
- **Studio.** Four Office formats break the builder rule, podcasts are WAV only, and deleting an artifact through the documents route leaves its files behind. See [Studio](architecture/studio.md).
|
||||
- **Studio.** PowerPoint and Excel break the builder rule: their model-written scripts run in the script runner, but nothing confines a script's network or file access, and they keep no spec, so they have no versions or Refine. Podcasts are WAV only. See [Studio](architecture/studio.md).
|
||||
- **Local models follow-ups.** What the llama.cpp runtime left open: bugs in fit and fingerprinting, install screen gaps, adding a `.gguf` from disk, and constrained decoding. See [local models](architecture/local-models/runtime.md).
|
||||
- **Desktop app gaps.** Small, bounded gaps in documents, freshness and license settings, plus folders, which need a design first. See [documents](architecture/documents.md).
|
||||
- **Desktop app gaps.** Small, bounded gaps in documents, freshness and license settings. Library folders shipped in 2.2.0; their Trash and linking a folder on disk are designed in [01](proposals/file-agent/01-sources-and-folders.md) and not built. See [documents](architecture/documents.md).
|
||||
- **Chat eval.** Score the chat's answers on every curated chat model: up to 8B on the real files on our own machines, 8B to 32B on Featherless, with 8B in both places as the check. See the [proposal](proposals/chat-eval.md).
|
||||
- **Retrieval: a multilingual embedder.** Ranking is done and measured; the remaining gap is that a question in one language does not find its answer in another, which no ranking change reaches. The swap is small and the re-embed migration is the work. See the [proposal](proposals/retrieval.md) and [search](architecture/search.md).
|
||||
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ The hosted Studio lets the model write code, runs it in a network-denied sandbox
|
|||
|
||||
## Where the code stands
|
||||
|
||||
- PPTX and XLSX do not follow the rule. Their prompts ask the model to "write one standalone Python script", and [`worker/studio/office/runner.py`](https://github.com/MODSetter/SurfSense/blob/0847e12f723394ff8343e586c074d6ebb3541fed/surfsense_local/backend/worker/studio/office/runner.py) runs the reply with `exec()` in the worker process, unsandboxed, bounded by a 120-second timeout that cannot hard-kill a runaway thread. The module's `ponytail:` comment names that ceiling, and an out-of-process sandbox runner as the upgrade path. For these two formats the risk is back at arbitrary code with the worker's privileges.
|
||||
- PPTX and XLSX do not follow the rule. Their prompts ask the model to "write one standalone Python script", which since [#2184](https://github.com/MODSetter/SurfSense/pull/2184) runs in the out-of-process runner of [ADR 0039](0039-document-scripts-run-without-approval.md), stopped at its time limit, rather than with `exec()` in the worker through [`worker/studio/office/runner.py`](https://github.com/MODSetter/SurfSense/blob/0847e12f723394ff8343e586c074d6ebb3541fed/surfsense_local/backend/worker/studio/office/runner.py), now deleted. For these two formats the risk is still arbitrary code with the user's privileges, in a process of its own that nothing confines ([studio](../architecture/studio.md#known-gaps)).
|
||||
- DOCX and PDF follow the rule on small models, which write Markdown for committed builders, and run a strong model's script in the out-of-process runner of [ADR 0039](0039-document-scripts-run-without-approval.md) ([studio](../architecture/studio.md#word-and-pdf)).
|
||||
- The other eight formats (summary, mind map, flashcards, quiz, HTML, podcast, image and infographic) execute nothing the model wrote.
|
||||
- The HTML viewer renders into `<iframe sandbox="allow-scripts allow-popups">` ([`features/studio/viewers/html-viewer.tsx`](../../surfsense_local/frontend/src/features/studio/viewers/html-viewer.tsx)).
|
||||
|
|
|
|||
|
|
@ -27,5 +27,5 @@ A command runs with the user's privileges, and a program it starts can open netw
|
|||
|
||||
## Where the code stands
|
||||
|
||||
- No agent exists yet.
|
||||
- The Office runner's 120-second limit is a `thread.join`, which cannot stop a thread that ignores it ([studio](../architecture/studio.md)).
|
||||
- The agent exists, with no shell ([ADR 0039](0039-document-scripts-run-without-approval.md)).
|
||||
- Since [#2184](https://github.com/MODSetter/SurfSense/pull/2184) no Studio format runs `exec()` in the worker: every Office script runs in ADR 0039's runner, a process stopped at its time limit ([studio](../architecture/studio.md)).
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ The user's sources are untrusted input, and a document can carry instructions ai
|
|||
|
||||
## Where the code stands
|
||||
|
||||
The runner and the agent's side are built on `dev_mod` ([the create-and-edit slice](../proposals/file-agent/07-create-and-edit-mvp.md)):
|
||||
The runner and the agent's side are built and shipped in 2.2.0 ([the create-and-edit slice](../proposals/file-agent/07-create-and-edit-mvp.md)):
|
||||
|
||||
- **The runner.** [`worker/document_script/`](../../surfsense_local/backend/worker/document_script/) runs a script in a process of its own: the frozen worker binary in its `--run-document-script` mode when packaged, [`worker.py`](../../surfsense_local/backend/worker.py) on the backend's Python in development. It gives the script an empty folder, one output path, the images the call names and an environment built from a few system variables, with no key or secret. At 120 seconds, or on a cancel, it kills the script and everything it started: a job object holds them on Windows, a process group elsewhere ([studio](../architecture/studio.md#script-documents)).
|
||||
- **Kept with its version.** An agent's Word document or PDF is a script document: its script is the artifact's spec, each edit is a new version, and the next edit reads and changes that script.
|
||||
|
|
@ -38,4 +38,4 @@ The runner and the agent's side are built on `dev_mod` ([the create-and-edit sli
|
|||
|
||||
- **Studio's Word and PDF.** A strong model's script runs through the runner and is kept as the version's spec; a small model writes Markdown that committed builders render ([studio](../architecture/studio.md#word-and-pdf)).
|
||||
|
||||
Not yet: Studio's PowerPoint and Excel still run with `exec()` in [`office/runner.py`](https://github.com/MODSetter/SurfSense/blob/0847e12f723394ff8343e586c074d6ebb3541fed/surfsense_local/backend/worker/studio/office/runner.py), and nothing blocks a script's network access or the processes it starts.
|
||||
Since [#2184](https://github.com/MODSetter/SurfSense/pull/2184), Studio's PowerPoint and Excel scripts run through the runner too, and [`office/runner.py`](https://github.com/MODSetter/SurfSense/blob/0847e12f723394ff8343e586c074d6ebb3541fed/surfsense_local/backend/worker/studio/office/runner.py) is deleted. Not yet: nothing blocks a script's network access or the processes it starts.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ A chat thread can be the agent's. opencode, started by Electron, then answers it
|
|||
| Part | Where | Does |
|
||||
|---|---|---|
|
||||
| Staging | `electron/scripts/opencode/` | `pnpm build:opencode` fetches opencode `1.18.34` and ripgrep `15.1.0`, each checked against its SHA-256, into `electron/opencode/` ([packaging](packaging.md)), for every build unless [`enabled.mjs`](../../surfsense_local/electron/scripts/opencode/enabled.mjs) is told otherwise; [`check-stage.mjs`](../../surfsense_local/electron/scripts/opencode/check-stage.mjs) runs both from the stage, electron-builder's [`after-sign.mjs`](../../surfsense_local/electron/scripts/opencode/after-sign.mjs) hook runs it on the signed copy before any installer is made, and release CI runs it on each installer's copy |
|
||||
| Process | `electron/src/main/sidecars/opencode.ts`, `opencode-home.ts`, `opencode-leftovers.ts`, `watchAgentConfig` in `index.ts` | starts, stops and cleans up `opencode serve` ([overview](overview.md)) |
|
||||
| Process | `electron/src/main/sidecars/opencode.ts`, `opencode-home.ts`, `opencode-leftovers.ts`, `watchAgentConfig` in `sidecars/boot.ts` | starts, stops and cleans up `opencode serve` ([overview](overview.md)) |
|
||||
| Engine choice | `backend/modules/agent/engine_choice.py` | whether a thread opened now is the agent's |
|
||||
| Configuration | `opencode_config.py`, `prompts/agent.md`, `skills/` | the file opencode runs with, and the skills it may load |
|
||||
| Readiness | `opencode_runtime.py`, `model_window.py`, `model_reads_images.py` | an opencode serving the current configuration |
|
||||
|
|
@ -130,7 +130,7 @@ Deleting a thread aborts and deletes its session, disposes its opencode instance
|
|||
| `surfsense_pdf_stamp` | `kind`, `watermark`, `page_numbers`, `header` or `footer`; `document_id` or `artifact_id`; `text`, with `{page}` and `{total}` for page numbers; `position`, one of six corners and centres; `pages`; `title` | a new PDF artifact with the text drawn upright on each page named, as the page is shown |
|
||||
| `surfsense_pdf_form` | `action`, `list` or `fill`; `document_id` or `artifact_id`; `fields`, an object of field name to value; `flatten`; `title` | `list`: each field's full name, kind, pages, value now and options. `fill`: a new PDF artifact with the fields set, or, flattened, with the values drawn into the pages and no fields left |
|
||||
|
||||
The schemas are flat, in a fixed order, and the same on every turn, and a test pins them. A call the tool cannot carry out comes back as a tool error the model reads: no query, no sources named, Studio's own reason such as "Needs a chat model", or a search before onboarding has chosen an embedder or while its files are missing, which points the model at `grep`. A source's own text loses any `passage` tag, so it cannot make up a label. A tool runs in one transaction off the event loop, except a tool that waits on another process or does slow file work (`waits` on [`tool.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/tool.py)), which commits its own short transactions so the write lock never stays held meanwhile: `surfsense_render_document`, `surfsense_revise_document`, `surfsense_list_images`, which reads each source's index once and copies the figures after its commit, `surfsense_source_pages`, which may wait on LibreOffice or Electron, `surfsense_convert_document`, which waits for its Studio job, `surfsense_analyze_data`, which waits on the Studio worker, and the three PDF tools, which read and write files. A tool returning images only (`needs_image_input`) is listed and run only for a model that reads images; a tool added later comes after it, so each model's list keeps its order from turn to turn. A result is one text item, then, for `surfsense_render_document`, `surfsense_revise_document`, `surfsense_convert_document` and `surfsense_source_pages`, one MCP image item per page, for `surfsense_analyze_data` one per chart, and for the PDF tools one per page shown (`{type: "image", data, mimeType: "image/jpeg"}`, a `ToolResult` in [`tool.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/tool.py)); an error is text only, since opencode drops an error's images.
|
||||
The schemas are flat, in a fixed order, and the same on every turn, and a test pins them. An optional integer `*_id` sent as 0 is dropped before the tool runs, since ids count from 1 ([`offered_tools.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py)): OpenAI's models fill every field a tool lists, so a render with no template asked for source 0, was refused, and was sent again until the turn ran out. In the OpenRouter screening no OpenAI model rendered a document before this, and the models it had failed were screened again ([model capabilities](model-capabilities.md#the-list)). A required id of 0 still reaches its tool, which says it is wrong. A call the tool cannot carry out comes back as a tool error the model reads: no query, no sources named, Studio's own reason such as "Needs a chat model", or a search before onboarding has chosen an embedder or while its files are missing, which points the model at `grep`. A source's own text loses any `passage` tag, so it cannot make up a label. A tool runs in one transaction off the event loop, except a tool that waits on another process or does slow file work (`waits` on [`tool.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/tool.py)), which commits its own short transactions so the write lock never stays held meanwhile: `surfsense_render_document`, `surfsense_revise_document`, `surfsense_list_images`, which reads each source's index once and copies the figures after its commit, `surfsense_source_pages`, which may wait on LibreOffice or Electron, `surfsense_convert_document`, which waits for its Studio job, `surfsense_analyze_data`, which waits on the Studio worker, and the three PDF tools, which read and write files. A tool returning images only (`needs_image_input`) is listed and run only for a model that reads images; a tool added later comes after it, so each model's list keeps its order from turn to turn. A result is one text item, then, for `surfsense_render_document`, `surfsense_revise_document`, `surfsense_convert_document` and `surfsense_source_pages`, one MCP image item per page, for `surfsense_analyze_data` one per chart, and for the PDF tools one per page shown (`{type: "image", data, mimeType: "image/jpeg"}`, a `ToolResult` in [`tool.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/tool.py)); an error is text only, since opencode drops an error's images.
|
||||
|
||||
Before each turn the API adds the tools to the thread's folder with opencode's `POST /mcp?directory=…`: the server `surfsense`, the thread's URL, the launch key as a header, OAuth off, and a call timeout of 200 seconds, where a call would otherwise get the MCP SDK's 60 ([`registration.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/registration.py)). opencode keeps it in memory for that folder's instance alone and drops it when it reloads, so it is added on every turn, nothing is written to `opencode.json`, which every folder shares, and two threads' turns at once each keep their own. opencode then lists the tools and names them `surfsense_<tool>`. When it cannot reach them, the turn runs with its own tools and the API logs why. The chat panel shows their steps as "Searched the sources for …", "Started … in Studio", "Created *Title* v1", "Revised *Title* v2", "Read the script behind a document", "Looked for images in 2 sources" and "Ran the analysis *Title*", or "Analysed data" while its title is still being written. A PDF tool's step says what it did, by its `operation`, `kind` or `action`: "Merged PDFs", "Split a PDF", "Numbered a PDF’s pages", "Filled in a PDF form" and so on, or "Worked on PDF pages" while its input is still being written.
|
||||
|
||||
|
|
@ -220,7 +220,7 @@ The thread routes in [chat](chat.md#agent-threads) hand an agent thread to `agen
|
|||
The chat panel sends the sources ticked in the sources panel with every message, and they bound an agent turn as they bound a chat's retrieval, resolved the same way in one transaction ([`turn_sources.py`](../../surfsense_local/backend/modules/agent/agent_threads/turn_sources.py)): a sent `source_scope` is stored on the thread; with none, `document_ids` is checked as a chat checks it (`load_selected_sources`, `422` for a source not in the workspace, `409` for one not ready) and stored as `{document_ids}`; with neither, the thread's stored scope holds, and a thread that never stored one uses every source. The scope resolves to its ready sources, with no cap, so a ticked folder of any size is the whole turn; every source means the workspace's files and notes and its filed artifacts, never an unfiled one ([source scope](chat.md#source-scope)). An empty result is no sources.
|
||||
|
||||
- **The folder holds it.** The thread's `sources/` is brought in line with the resolved sources before the turn, so `read`, `grep` and `glob` reach no other source ([folders](#folders)). A Studio artifact filed in a ticked folder is mirrored like any source.
|
||||
- **The model is told.** Unless the stored scope is every source, the message goes to opencode with a second text part marked `synthetic`, which opencode 1.18.34 accepts on `prompt_async`, keeps as not the user's, and passes to the model inside the user's message ([`scope_note.py`](../../surfsense_local/backend/modules/agent/agent_threads/scope_note.py)). Its first line is SurfSense's tag, `[surfsense-scope: 3,7]`, past 200 sources `[surfsense-scope: count=N]`, or `[surfsense-scope: none]`; the rest says the user chose N sources for this chat, to use only those, and that passages read earlier from other sources no longer apply, or that no sources are selected, or, when every ticked source is still being read, that none is ready yet. Up to 20 sources, it names each one's file by its path under `sources/`, as the thread's sync lays it out, and says to open it with `read` or search it with `surfsense_search_sources`: Qwen3 1.7B, told only that the source was "the file in `sources/`", guessed `sources/17.md` and gave up. Past 20, or when one ticked source has no text to mirror, it says they are the files in `sources/` and names none. Every note ends by saying SurfSense added it and it is never to be repeated, as a small model began its answer with the tag. It lives in the session, so a compaction keeps it, and each turn carries the ticks of the moment it was sent.
|
||||
- **The model is told.** Unless the stored scope is every source, the message goes to opencode with a second text part marked `synthetic`, which opencode 1.18.34 accepts on `prompt_async`, keeps as not the user's, and passes to the model inside the user's message ([`scope_note.py`](../../surfsense_local/backend/modules/agent/agent_threads/scope_note.py)). Its first line is SurfSense's tag, `[surfsense-scope: 3,7]`, past 200 sources `[surfsense-scope: count=N]`, or `[surfsense-scope: none]`; the rest says the user chose N sources for this chat, to use only those, and that passages read earlier from other sources no longer apply, or that no sources are selected, or, when every ticked source is still being read, that none is ready yet. Up to 20 sources, it names each one's file by its path under `sources/`, as the thread's sync lays it out, in double quotes, which a mirrored name never holds though it can hold a backtick, and says to open it with `read` or search it with `surfsense_search_sources`: Qwen3 1.7B, told only that the source was "the file in `sources/`", guessed `sources/17.md` and gave up. Told the path, asked about Boston's CM11 contract, Qwen3 1.7B (UD-Q4_K_XL, on llama.cpp) read the file and answered from its clause 5.2, and Qwen3 4B (Q2_K) and Qwen3 8B searched it and answered. Past 20, or when one ticked source has no text to mirror, it says they are the files in `sources/` and names none. Every note ends by saying SurfSense added it and it is never to be repeated, as a small model began its answer with the tag. It lives in the session, so a compaction keeps it, and each turn carries the ticks of the moment it was sent.
|
||||
- **The tools hold to it.** Each thread registers the tools at its own address, and a tool call resolves the thread's stored scope in one short transaction ([`turn_scope.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/turn_scope.py)), so ticks stored between turns hold the next call. `surfsense_search_sources` searches only those sources, and with none says so instead; `surfsense_list_images`, `surfsense_source_pages`, `surfsense_create_artifact` and the PDF tools refuse a source outside them, naming at most 20 of the sources that are in it and then "and N more: the files in `sources/`"; `surfsense_render_document` refuses an image or a template from one, including the template a next version would keep from its base, and places an analysis chart only from its own thread; `surfsense_revise_document` and `surfsense_analyze_data` refuse a source outside them. An address naming no agent thread of the workspace is refused rather than widened. Artifacts the agent rendered are its outputs, so `surfsense_read_document` needs no scope, nor does a render that continues one, beyond its template.
|
||||
- **The thread shows it.** Reading the thread back takes the note out of the user's text and returns `content.scope`: `{document_ids, titles}`, keeping the sources that still exist, or for a counted tag `{document_ids: [], titles: [], count}`. Under the user's message the screen shows "Working from Plan.pdf", up to three names inline, else "Working from 5 sources" with the names on hover, "Working from 5,000 sources" with nothing on hover for a counted tag, or "Working from no sources": live from the `agent-scope` frame that follows `agent-preparing`, which carries the same shape, and on read-back from the stored scope ([`working-from.tsx`](../../surfsense_local/frontend/src/features/agent/working-from.tsx)). A turn on every source, and every chat thread, shows none.
|
||||
|
||||
|
|
@ -237,7 +237,7 @@ A document script is another matter: it runs in the Studio worker's runner, whic
|
|||
## Tests
|
||||
|
||||
- **Backend:** [`tests/integration/agent/`](../../surfsense_local/backend/tests/integration/agent/) and [`tests/unit/agent/`](../../surfsense_local/backend/tests/unit/agent/). The client, readiness and thread tests start the staged opencode against a scripted model, and the thread tests also run the real API on a port. The tool endpoint tests drive the app in-process, as opencode's MCP client calls it. They skip where `pnpm build:opencode` has not staged a build. Each builds the app as Electron starts it beside an opencode; `test_routes_without_opencode.py` builds it without one. `test_document_flow.py` plays a whole document flow through the tool endpoint with no model: a logo's figure, Word v1 with the logo and a chart, its script read back, v2, and a PDF whose previews come back as images. `test_office_documents.py` renders decks, workbooks and documents made from a source's template through the real runner, with Electron played on the snapshot queue for a deck's slides, `test_source_pages.py` draws sources' pages with Electron played on the snapshot queue and LibreOffice by a stand-in (`tests/office_stand_in.py`; every test starts with Office support off), `test_convert_document.py` converts versions and sources through the stand-in with Studio's worker on a thread, `integration/office_support/test_libreoffice.py` (marker `office`) converts a Word file and recalculates a workbook on the LibreOffice installed on the machine, named explicitly, or skips, `test_documents_skill.py` and `test_data_skill.py` run the skills' examples through the real runner, `test_analyze_data.py` runs analyses on the Studio worker played on a thread and places a chart in a Word file, and `test_compacted_turns.py` reads compacted sessions back. `test_tool_scope.py` holds each tool to a turn's ticked sources in-process, `test_thread_folders.py` lays out, links and sweeps thread folders, `test_thread_isolation.py` runs two threads in the real opencode and checks neither reads, `grep`s nor `glob`s the other's sources, `unit/agent/test_mirror_path.py` checks the mirror's names and length cuts, and `test_agent_scope.py` sends ticked, empty and omitted selections, a ticked folder past 1,000 sources and a thread's stored scope through the real opencode. `test_pdf_tools.py` calls the PDF tools in-process over uploaded PDFs and artifacts, `test_pdf_skill.py` checks the PDF skill against the tools and the range parser, [`tests/unit/pdf_tools/`](../../surfsense_local/backend/tests/unit/pdf_tools/) covers each operation, stamp and form on PDFs built with ReportLab, and `tests/integration/artifacts/test_made_files.py` takes a made file through Studio's job.
|
||||
- **Live:** [`tests/live/`](../../surfsense_local/backend/tests/live/) drives the real path against Claude Sonnet on a spend budget, skipped unless asked for ([07-dev-setup](../proposals/file-agent/07-dev-setup.md#live-tests)).
|
||||
- **Live:** [`tests/live/`](../../surfsense_local/backend/tests/live/) drives the real path against Claude Sonnet on a spend budget, skipped unless asked for ([07-dev-setup](../proposals/file-agent/07-dev-setup.md#live-tests)). [`tests/live/sweep/`](../../surfsense_local/backend/tests/live/sweep/) runs its smoke, PDF brief and board pack cases over OpenRouter's tool-calling models for the [capability list](model-capabilities.md#the-list).
|
||||
- **Electron:** the sidecar tests in [`electron/src/main/sidecars/`](../../surfsense_local/electron/src/main/sidecars/), the snapshot poller, session, print-option and page-answer tests in `electron/src/main/docx-snapshot/`, and the pins, switch and stage-check tests in `electron/scripts/opencode/`.
|
||||
- **Frontend:** [`features/agent/agent-thread.test.tsx`](../../surfsense_local/frontend/src/features/agent/agent-thread.test.tsx), on the real dashboard against a stream that waits for the test's answer, a legacy thread's history and refusal in `outdated-agent-thread.test.tsx`, the Working-from line's cases, the counted one included, in `working-from.test.tsx`, the step labels in `agent-steps.test.tsx`, and the snapshot page's Word and deck layout, altChunk and policy tests in `features/docx-snapshot/`.
|
||||
|
||||
|
|
@ -259,9 +259,9 @@ A document script is another matter: it runs in the Studio worker's runner, whic
|
|||
- On a disk without hard links each thread holds its own copy of each text, and pays Defender's scan again.
|
||||
- A configuration rewrite, which a change of model, window or image input causes, ends every running agent turn in every workspace.
|
||||
- Opening the first agent thread waits for opencode to start and for a local model to load, up to a minute, with nothing on screen but the thread being created.
|
||||
- When SurfSense's data folder sits inside a git repository, such as a home folder kept in git, opencode counts the repository's root as its project, not the thread's folder. Then every thread's agent can read, `grep` and `glob` every other thread's folder, including sources not ticked for it, and every other workspace's, without asking. Nothing detects it.
|
||||
- When SurfSense's data folder sits inside a git repository, such as a home folder kept in git, opencode counts the repository's root as its project, not the thread's folder. Then every thread's agent can read, `grep` and `glob` every other thread's folder, including sources not ticked for it, and every other workspace's, without asking. Nothing detects it ([#2229](https://github.com/MODSetter/SurfSense/issues/2229)).
|
||||
- No test runs opencode with every connection beyond loopback refused to check that it opens none.
|
||||
- No released installer has carried opencode yet: a notarized build on macOS and a signed build under Defender on Windows are untried until the first release's CI runs each installer's copy ([packaging](packaging.md#release-builds)).
|
||||
- When opencode cannot stay up, each chat opened in Agentic waits the full 60 seconds before its `503`, and Electron starts opencode again every 10 seconds while its configuration stands, each exit showing the generic "A background service stopped" toast again, which names neither the agent nor the cause. A failure to spawn it at all is not reported as a crash ([overview](overview.md#known-gaps)) ([#2228](https://github.com/MODSetter/SurfSense/issues/2228)).
|
||||
- opencode's tool-output folder is shared by every thread in every workspace and open to each one's agent: a tool result over 50 KB or 2,000 lines is kept there for 7 days (SurfSense's own tools stay under that limit; `surfsense_read_document` pages long scripts), and another thread's agent, in the same workspace or another, can list and read it without asking. Per-thread folders do not change this.
|
||||
- On macOS and Linux, opencode checks an absolute path as written, so its own allow for its tool-output folder also matches a path that climbs out of it with `..`. The skills folder's allow has a `..` deny after it; the tool-output folder's cannot, since opencode adds that allow after SurfSense's rules. Windows normalises the path first.
|
||||
- Every page image the agent is shown, each render's previews and each page it opens, stays in its context until a compaction, and Anthropic's OpenAI-compatible API reports no prompt caching, so each request bills them again: about 1,000 tokens a page a request. Nothing drops the images of a version a newer one superseded. Anthropic caps the images in one request, so about 25 four-page renders in one uncompacted thread could reach it.
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ A chat thread belongs to a workspace. Each user message retrieves its own contex
|
|||
|
||||
| Method | Path | Does |
|
||||
|---|---|---|
|
||||
| `POST` | `/workspaces/{workspace_id}/chat/threads` | open a thread, with an optional `source_scope` and `mode`, `basic` or `agentic` ([Agent threads](#agent-threads)); the title defaults to "New chat" |
|
||||
| `POST` | `/workspaces/{workspace_id}/chat/threads` | open a thread, with an optional `source_scope`, `mode`, `basic` or `agentic`, and `remember`, which makes the mode the model's default ([Agent threads](#agent-threads)); the title defaults to "New chat" |
|
||||
| `GET` | `/workspaces/{workspace_id}/chat/threads` | list, newest first, each with `running` while a reply is being generated for it, and `run_state`, `{state, position}`: `queued` with its place in line, `running`, or `needs-approval` while an agent turn waits on the user; `null` when nothing runs |
|
||||
| `PATCH` | `/chat/threads/{thread_id}` | rename, 1 to 200 characters |
|
||||
| `DELETE` | `/chat/threads/{thread_id}` | delete; a running reply is stopped first, waiting up to 5 seconds, then its messages cascade, its artifacts stay with the thread cleared, and its images' folder goes after the commit |
|
||||
|
|
@ -182,7 +182,8 @@ What it does not do:
|
|||
## Frontend runtime
|
||||
|
||||
- [`use-chat-runtime.ts`](../../surfsense_local/frontend/src/features/chat/use-chat-runtime.ts) holds the threads, the open thread's stored turns and the actions. The thread panel hands them to [`live-thread-runtime.tsx`](../../surfsense_local/frontend/src/features/chat/live-thread-runtime.tsx), which adds the open thread's run and builds assistant-ui's `useExternalStoreRuntime`. FastAPI is already the source of truth for threads and messages, and a local-runtime history adapter would add a second persistence path that could write a message twice.
|
||||
- A streamed token renders the open thread's live reply and nothing else. The page reads each run's summary only (its state, whether it ended, whether it has its pair), which keeps its identity while text arrives. The runtime host reads the run and is handed the panel's content already built, so a token renders neither the panel nor the sidebar and Studio. The message renderer, the citation context and each turn's conversion hold still between frames, so a finished reply is not converted or rendered again. The run store tells subscribers of text at most once per macrotask and no sooner than 32 ms after its last notice (`TEXT_NOTICE_GAP_MS`), so the frames of one network read, a reattach's replay among them, render once, and a fast model's reply renders about 30 times a second however long it gets; not once per animation frame, which a hidden window never gets. A run's start, end, stop and drop are told at once, before its end is announced. A frame that changes nothing the run shows, such as a title or an approval, is told to no subscriber; frame listeners still hear every frame.
|
||||
- A streamed token renders the open thread's live reply and nothing else. The page reads each run's summary only, `runSummaries()` (its state, whether it ended, whether it has its pair), which keeps its identity while text arrives. The runtime host, `LiveThreadRuntime`, reads the run and is handed the panel's content already built, so a token renders neither the panel nor the sidebar and Studio. The message renderer, the citation context and each turn's conversion hold still between frames, so a finished reply is not converted or rendered again. Measured in the dev build, a token costs about 24 ms in a thread of 22 messages and 21 ms beside 1,000 sources; when every frame of any run, a background thread's included, rendered the whole page and parsed each message's markdown again, it cost 538 ms and 3.6 s. The run store tells subscribers of text at most once per macrotask and no sooner than 32 ms after its last notice (`TEXT_NOTICE_GAP_MS`): `notifySoon()` sets one timer, due 32 ms after the last notice, or on the next macrotask after a quiet spell. Each render reads the whole live reply, and a notice per network read kept the window busy redrawing a fast model's long one. So the frames of one network read, a reattach's replay among them, render once, and a fast model's reply renders about 30 times a second however long it gets; not once per animation frame, which a hidden window never gets. A run's start, end, stop and drop are told at once, before its end is announced. A frame that changes nothing the run shows, such as a title or an approval, is told to no subscriber; frame listeners still hear every frame.
|
||||
- The columns around the thread render only when what they show changes. The left sidebar, the sources panel, its tree and each row, Studio, its format cards, the artifact list and each row, and the Chats dialog are memoized, and the dashboard hands them values and callbacks that keep their identity, a handler that reads changing state through [`useStableCallback`](../../surfsense_local/frontend/src/hooks/use-stable-callback.ts). The Chats dialog builds its rows only while it is open, and a re-read of the sources list keeps its unchanged rows ([`reuse-unchanged.ts`](../../surfsense_local/frontend/src/features/sources/reuse-unchanged.ts)). Two tests count renders: [`dashboard-streaming.test.tsx`](../../surfsense_local/frontend/src/features/dashboard/dashboard-streaming.test.tsx) fails when a streamed token renders the sidebar, the sources panel, Studio, the thread panel or a reply's message, and [`dashboard-renders.test.tsx`](../../surfsense_local/frontend/src/features/dashboard/dashboard-renders.test.tsx) when a token renders a source row, an artifact row or Studio, a column drag renders anything inside the columns, or opening a source preview renders a source row.
|
||||
- Threads and messages are TanStack Query data. Replies in progress live outside React, in a store of one run per thread ([`runs/run-store.ts`](../../surfsense_local/frontend/src/features/chat/runs/run-store.ts)), so leaving a thread drops nothing: its reply keeps arriving in the background. The view is the stored turns with the run's live pair in place of their copies: an optimistic user turn and a draft assistant turn until `accepted` swaps in the real ids. One pure fold builds the pair from frames, whether they arrive from the send or from a replay ([`runs/apply-frame.ts`](../../surfsense_local/frontend/src/features/chat/runs/apply-frame.ts)). assistant-ui keeps every message id it is given, so the optimistic user turn is marked optimistic: it, and the picked images it holds, go when `accepted` renames it. When a run ends the thread is refetched, and the live pair is dropped once the stored turns contain both ids; a thread that is not open is read once more if the first read lags, so its run does not wait for the thread to be opened. Only the run that ended is dropped: a Retry pressed while that read is out is a new run, and the read leaves it streaming.
|
||||
- Opening a thread the list marks `running` that this window does not follow, after a reload or from another window, follows its run from frame 0 and rebuilds the reply from the replay; the question is the stored turn's. A `chat-runs` event refreshes the list; its `done` refreshes the open thread's turns, or marks another thread unread.
|
||||
- A reply that ends while another thread is open marks its thread unread: a dot on its row in the Chats dialog and on the sidebar's Chats button, until the thread is opened. The set is per workspace in `localStorage` under `surfsense:unread-replies:<workspace>:v1`, and a run carries its workspace, so a reply that ends after another workspace was opened is marked in its own.
|
||||
|
|
@ -190,7 +191,7 @@ What it does not do:
|
|||
- Sending from a new chat first creates a thread titled "New chat", in the mode the composer's switch shows ([model capabilities](model-capabilities.md#the-modes)). When that is refused, the question and its images go back into the composer. The last open thread of each workspace is remembered in `localStorage`.
|
||||
- [`sse.ts`](../../surfsense_local/frontend/src/features/chat/sse.ts) parses the stream chunk-safely, with each frame's number: frames split on blank lines, and a partial frame waits for the next read, searched only where a read added to it, so a large frame costs its length once rather than once per read. This token stream is separate from the `/events` channel.
|
||||
- From send until the first token, the assistant turn shows a shimmering "Thinking" beside an orbiting-dots indicator ported from surfsense_web ([`thinking-indicator.tsx`](../../surfsense_local/frontend/src/features/chat/thinking-indicator.tsx)), so a model load or a long prompt is never a blank reply. It is one header from send to answer, whose state moves from waiting to thinking to done, so the indicator and shimmer never restart when the trace arrives. While a local reply waits for admission the label is "Waiting for another reply (2nd in line)", from the latest `run-state`. While the local runtime reads a long prompt the label is "Reading 42%" instead, from the latest `prompt-progress` frame, shown only between the first token read and the last, so a prompt of one batch and a turn with no such frame stay "Thinking". The header's `role="status"` region follows by quarters ("Reading 25%"), not at every frame. The figure is kept with the live message only and is never stored. A thinking model's trace then streams open below it, above the answer, rendered as markdown by the same Streamdown setup and link and image restrictions as the answer, in a box 13rem tall that fades at its scrolled edges and follows the newest line until the reader scrolls up, and folds to "Thought for N seconds" when the answer starts, the indicator sliding away; the person's own open or close wins until the next reply, and holds the chat's scroll still while the trace slides, with assistant-ui's `useScrollLock` ([`reply-thinking.tsx`](../../surfsense_local/frontend/src/features/chat/reply-thinking.tsx)). The trace rides in the message's `metadata.custom`, beside the citations, not as a content part, so Copy takes the answer alone.
|
||||
- Replies, the trace, and Studio's Markdown documents and podcast transcripts render through one Streamdown setup ([`streamdown-config.ts`](../../surfsense_local/frontend/src/features/studio/viewers/streamdown-config.ts)) whose options keep their identity between renders, so a token re-parses only the block it changed and a page re-render around a finished reply or an open artifact parses nothing. The trace repairs an unclosed marker at its tail only, as the answer does, not across the whole trace at every token, so a marker left open in an earlier paragraph stays as written instead of closing at the trace's end. The live trace scrolls to its newest line at most once a frame, and only while open; the follow's own scroll, reported a frame later, never counts as the reader scrolling up. Code stays plain monospace while its reply or trace streams and takes its colours when that ends: highlighting each partial state of a growing block cost the main thread seconds and kept every state's tokens until reload. The highlighter ([`code-highlighter.ts`](../../surfsense_local/frontend/src/features/studio/viewers/code-highlighter.ts)) is @streamdown/code's, Shiki with the same themes, languages and tokens, keeping those of the last 100 blocks.
|
||||
- Replies, the trace, and Studio's Markdown documents and podcast transcripts render through one Streamdown setup ([`streamdown-config.ts`](../../surfsense_local/frontend/src/features/studio/viewers/streamdown-config.ts)) whose options keep their identity between renders, so a token re-parses only the block it changed and a page re-render around a finished reply or an open artifact parses nothing. The trace repairs an unclosed marker at its tail only, as the answer does, not across the whole trace at every token, so a marker left open in an earlier paragraph stays as written instead of closing at the trace's end. The live trace scrolls to its newest line at most once a frame, and only while open; the follow's own scroll, reported a frame later, never counts as the reader scrolling up. Code stays plain monospace while its reply or trace streams and takes its colours when that ends: highlighting each partial state of a growing block cost the main thread seconds and kept every state's tokens until reload, a 4,000-character block streamed 4 characters at a time making 995 highlight calls and holding 80 MB after garbage collection. The highlighter ([`code-highlighter.ts`](../../surfsense_local/frontend/src/features/studio/viewers/code-highlighter.ts)) is @streamdown/code's, Shiki with the same themes, languages and tokens, keeping those of the last 100 blocks. A block it has not tokenized yet comes back as plain code at once, never as nothing: given nothing, Streamdown keeps its last result, which for a block whose text just changed is older text.
|
||||
- Images use assistant-ui's own attachments ([`image-attachments.ts`](../../surfsense_local/frontend/src/features/chat/image-attachments.ts)). The runtime gets `SimpleImageAttachmentAdapter`, narrowed to the accepted formats, only while the selection's `reads_images` is true, so a model that cannot see has no adapter and the composer takes no image. The composer's "+" opens a menu ([`composer-add-menu.tsx`](../../surfsense_local/frontend/src/features/chat/composer-add-menu.tsx)) with separate "Attach images" and "Upload sources" items, plus the Thinking switch, so a file is never a guess between the two, and dropping a file onto the composer does nothing. "Upload sources" opens the same picker and upload as the sources panel's Add, and is disabled while an upload runs. Images come in two ways: pasting into the composer (`Input`'s `addAttachmentOnPaste`), and "Attach images", which is assistant-ui's `ComposerPrimitive.AddAttachment` and opens a picker limited to the adapter's formats. With a model that cannot see, the item stays in the menu, disabled but still hoverable, and its tooltip says "This model can’t read images". An Agentic chat, new or open, takes no image either, since the agent refuses a turn that carries one: attach and paste are off and the tooltip says "Agentic mode doesn’t read images yet". Images attached before a new chat was switched to Agentic stay in view to be removed, and sending them opens nothing: the question and images go back into the composer with a toast. Each row's tooltip is also in the row as screen-reader text, since Base UI tooltips are visual only ([`menu-item-hint.tsx`](../../surfsense_local/frontend/src/features/chat/menu-item-hint.tsx)); `ComposerPrimitive.Attachments` draws the pending ones with `AttachmentPrimitive` and `MessagePrimitive.Attachments` the sent ones ([`attached-image.tsx`](../../surfsense_local/frontend/src/features/chat/attached-image.tsx)). Until a turn is stored its images show from what was picked; after, from the image route. Retry resends a failed turn's images, read back from the image route once the window no longer holds them.
|
||||
- Sending is disabled without a usable model or while threads or messages load, and in a thread whose reply is still running. Stop calls `run/stop`, in either engine. A `409` "no chat model selected" opens model setup. A reply's error and stopped mark come from its stored `ending`, and from live frames only while its run is followed, so both survive a thread switch and a reload: auth, not-found and model-cannot-run errors, and network errors from a remote endpoint, offer Model setup; a network error from the local runtime, a context-too-long error and a used-up plan offer nothing, since no button fixes them; the rest, and an interrupted reply ("Interrupted when the app closed."), offer Retry, on the latest reply only, which sends the question again with `retry_of`.
|
||||
- Every turn sends the sources panel's ticks as `source_scope`, so a ticked folder counts whole however many sources it holds, and beside it the ids of the ready sources those ticks include, which the server ignores when a scope comes with them; past the API's 1,000-id cap only the scope goes. Unticking a source takes it out of retrieval, and unticking all of them leaves the model with no context.
|
||||
|
|
|
|||
|
|
@ -140,7 +140,7 @@ An artifact's searchable body is a `Document` with `document_type = ARTIFACT`; `
|
|||
| Table | Columns | Notes |
|
||||
|---|---|---|
|
||||
| `provider_connections` | `id`, `label`, `provider`, `base_url`, `catalog_provider`, `api_key_ciphertext`, timestamps, and since `0023` `auth_kind`, `oauth_ciphertext` and `token_version` | `label` unique case-insensitively; `provider` must be `openai_compatible`; `catalog_provider` is a remote manifest provider id or `custom`, since `0014`; the key is Fernet ciphertext since `0007`; since `0023`, `auth_kind` is `api_key` or `chatgpt`, `oauth_ciphertext` the Fernet-encrypted token set of a `chatgpt` row (`NULL` when signed out), and `token_version` the counter that keeps refreshes from racing ([`chatgpt-subscription.md`](chatgpt-subscription.md)) |
|
||||
| `selected_models` | `model_type`, `provider`, `connection_id`, `name`, `params_b`, `vendor`, `line`, `settings`, `updated_at` | one row per model type: `text_gen`, `image_gen`, `image_edit`, `video_gen` or `audio_gen`; `settings` is JSON for what the user set that no endpoint states, keyed by the slice that owns each entry, and cleared when the slot takes another model |
|
||||
| `selected_models` | `model_type`, `provider`, `connection_id`, `name`, `params_b`, `vendor`, `line`, `settings`, `updated_at` | one row per model type: `text_gen`, `image_gen`, `image_edit`, `video_gen` or `audio_gen`; `settings` is JSON for what the user set that no endpoint states, keyed by the slice that owns each entry, and cleared when the slot takes another model, all but `chat_modes`: the mode last picked for each model, keyed `<provider>/<connection id>/<name>`, which stays so a model's pick comes back with it ([model capabilities](model-capabilities.md#the-modes)) |
|
||||
| `onboarding_completion` | `id`, `completed_at` | a singleton (`CHECK id = 1`) whose presence means onboarding is done |
|
||||
|
||||
- A CHECK on `selected_models` allows `llamacpp`, `sdcpp` and `audiocpp` only without a connection and `openai_compatible` only with one, and a second, `local_runtime_type`, lets `llamacpp` hold only `text_gen`, `sdcpp` only `image_gen`, `image_edit` and `video_gen`, and `audiocpp` only `audio_gen`. `connection_id` cascades, so deleting a connection clears exactly the selections that used it.
|
||||
|
|
@ -388,3 +388,5 @@ erDiagram
|
|||
## Known gaps
|
||||
|
||||
- Deleting a folder is permanent: the Trash is designed, not built ([proposal](../proposals/file-agent/01-sources-and-folders.md)).
|
||||
- The snapshot needs free disk about the size of `surfsense.db`, and nothing checks for it first: when the copy fails, the API does not start and the app closes with no message ([overview](overview.md#known-gaps)).
|
||||
- A release cannot open a database a later release migrated, since its migrations do not know the newer revision, so its API does not start. Going back a version means restoring the snapshot by hand, as going from 2.2.0 to 2.1.0 takes `backups/0023-0027.db`.
|
||||
|
|
|
|||
|
|
@ -62,7 +62,7 @@ A note may carry `document_metadata`, stored as given. A plugin names itself the
|
|||
|
||||
The sources panel shows the Library as a tree ([`features/sources/tree/`](../../surfsense_local/frontend/src/features/sources/tree)). It reads every folder with `GET /folders` and every file and note page by page, 200 at a time, and builds the tree on the client; the Library's own folder is never a row, so a workspace with no folders is the flat list it always was.
|
||||
|
||||
- **Rows.** Folders come first, by name, then sources newest first. The list is an ARIA `tree` with one `treeitem` per row: arrow keys move and open, Space ticks, Enter opens, F2 renames and Delete asks to delete. A name filter, opened from the search button in the header, shows matches with the folders holding them opened; Escape or closing it clears it.
|
||||
- **Rows.** Folders come first, by name, then sources newest first. The list is an ARIA `tree` with one `treeitem` per row: arrow keys move and open, Space ticks, Enter opens, F2 renames and Delete asks to delete. A name filter, opened from the search button in the header, shows matches with the folders holding them opened; Escape or closing it clears it. A name the column cuts off shows whole in a tooltip on hover, or while its row has keyboard focus ([`overflow-tooltip.tsx`](../../surfsense_local/frontend/src/components/ui/overflow-tooltip.tsx)). Rows are memoized, and a re-read of the list keeps the object of every unchanged source ([`reuse-unchanged.ts`](../../surfsense_local/frontend/src/features/sources/reuse-unchanged.ts)), so a poll or an event renders only the rows whose source changed.
|
||||
- **Actions.** New folder (from the header's **+** menu, or inside a folder from its menu), Rename, Move to… and Delete. Move to… is a picker of folders and the keyboard's path; a row dragged onto a folder, or onto a source inside one, moves there too. A folder cannot be moved inside itself. Deleting a folder names how many sources, and filed Studio outputs, go with it, as `GET /folders/{folder_id}/summary` counts them, and deletes them for good.
|
||||
- **Ticks.** Each row's tick is tri-state. Ticking a folder ticks everything under it, now and added later; the nearest tick up the tree decides ([`scope-state.ts`](../../surfsense_local/frontend/src/features/sources/tree/scope-state.ts)). Chat and Studio send the ticks as `source_scope`, so every ticked source counts, not only the ones loaded.
|
||||
|
||||
|
|
|
|||
|
|
@ -39,7 +39,7 @@ of them.
|
|||
`GET /llm/catalog/local` returns llama.cpp's rows, then sd.cpp's, audio.cpp's and onnxruntime's.
|
||||
sd.cpp's are the six curated image models and two video models, only when Electron handed the API an
|
||||
images folder, which it does only when it staged sd-server, in dev or packaged
|
||||
([`index.ts`](../../../surfsense_local/electron/src/main/index.ts); [packaging](../packaging.md)).
|
||||
([`sidecars/boot.ts`](../../../surfsense_local/electron/src/main/sidecars/boot.ts); [packaging](../packaging.md)).
|
||||
audio.cpp's are the three curated audio models, only when Electron handed the API
|
||||
an audio folder, which it does only when it staged audio.cpp's server.
|
||||
onnxruntime's are the curated embedders, then any Hugging Face embedder
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ arguments each model gets, and every byte downloaded from Hugging Face. The
|
|||
shaping rule is to assume nothing worked until something says it did, because
|
||||
llama.cpp's common failures exit 0 and look like success.
|
||||
|
||||
**Code:** [`surfsense_local/backend/modules/llm/providers/llamacpp/`](../../../surfsense_local/backend/modules/llm/providers/llamacpp/), [`surfsense_local/electron/src/main/sidecars/llamacpp.ts`](../../../surfsense_local/electron/src/main/sidecars/llamacpp.ts), [`surfsense_local/electron/src/main/index.ts`](../../../surfsense_local/electron/src/main/index.ts) (preset watcher), [`surfsense_local/electron/scripts/fetch-llamacpp.mjs`](../../../surfsense_local/electron/scripts/fetch-llamacpp.mjs)
|
||||
**Code:** [`surfsense_local/backend/modules/llm/providers/llamacpp/`](../../../surfsense_local/backend/modules/llm/providers/llamacpp/), [`surfsense_local/electron/src/main/sidecars/llamacpp.ts`](../../../surfsense_local/electron/src/main/sidecars/llamacpp.ts), [`surfsense_local/electron/src/main/sidecars/watch-generation-preset.ts`](../../../surfsense_local/electron/src/main/sidecars/watch-generation-preset.ts) (preset watcher), [`surfsense_local/electron/scripts/fetch-llamacpp.mjs`](../../../surfsense_local/electron/scripts/fetch-llamacpp.mjs)
|
||||
**Decisions:** [ADR 0011](../../adr/0011-llama-cpp-local-runtime.md), [ADR 0012](../../adr/0012-vulkan-only-gpu-backend.md), [ADR 0015](../../adr/0015-openai-compatible-connections.md), [ADR 0049](../../adr/0049-prompts-grow-at-the-end.md)
|
||||
|
||||
How each model's window and cache precision are chosen is in
|
||||
|
|
@ -40,7 +40,7 @@ Router mode is what lets the sidecar boot on a clean machine. Measured at
|
|||
returns `{"data":[],"object":"list"}` against an empty directory. A router with
|
||||
three models discovered leaves VRAM unchanged from idle, in the plan's router
|
||||
lifecycle measurements, which do not record the platform. The directory itself
|
||||
must exist: a comment in `electron/src/main/index.ts` records that `llama-server`
|
||||
must exist: a comment in `electron/src/main/sidecars/boot.ts` records that `llama-server`
|
||||
exits 1 when `--models-dir` does not exist, so Electron creates it before the
|
||||
first start.
|
||||
|
||||
|
|
@ -131,7 +131,7 @@ the app never ships. When the binary is not staged the sidecar does not start.
|
|||
Boot waits only on the API's health; the runtime's state shows through
|
||||
`GET /llm/providers`.
|
||||
|
||||
`watchGenerationPreset()` in `electron/src/main/index.ts` polls the preset's size
|
||||
`watchGenerationPreset()` in `electron/src/main/sidecars/watch-generation-preset.ts` polls the preset's size
|
||||
and mtime every 5 s and restarts the sidecar on a change, the same shape as
|
||||
`watchImageModel()` for sd-server: the API is the authority, and a change is
|
||||
user-initiated and rare.
|
||||
|
|
|
|||
|
|
@ -83,6 +83,8 @@ Vite aliases `@formatjs/icu-messageformat-parser` to its no-parser build, since
|
|||
|
||||
Developers write English only, inline in the `formatMessage` call; `pnpm translations` or the pre-commit hook updates `en.json`. The [`translate`](../../.agents/skills/translate/SKILL.md) skill drafts Japanese and German for every id that is new or whose English changed, with its glossary and tone (です/ます; German *du*). Anyone who reads a language can correct it in a pull request; no release waits on review. No machine-translation service is used ([ADR 0017](../adr/0017-egress-off-by-default.md)).
|
||||
|
||||
The ten `README.<locale>.md` files at the repo root are not built from these catalogs, but they quote some of their strings, such as Settings or Office support; renaming one means changing them too ([06-repo-readme](../../plans/community-local/seo/06-repo-readme.md#the-readme-since-9-oct-2026)).
|
||||
|
||||
## Known gaps
|
||||
|
||||
- Backend prose without a code stays English in every language: the one refusal of a Hugging Face embedder repo that names a file ("Hugging Face lists no checksum for …"), an embedder's failed check at install, and Studio's refusals other than a missing model, such as "already generating" and "nothing is running", which the panel shows under its translated alert title.
|
||||
|
|
|
|||
|
|
@ -19,13 +19,13 @@ The bar, in [`verdict.py`](../../surfsense_local/backend/scripts/capability_list
|
|||
|
||||
[`measured/capabilities.json`](../../surfsense_local/backend/modules/llm/capability/measured/capabilities.json), schema version 1 ([`schema.py`](../../surfsense_local/backend/modules/llm/capability/measured/schema.py)), is read once per process and never fetched; a list that fails to load makes every model `not_measured`. Each row holds the canonical key, its matching rules (`match.keys`, and `match.served`: where the model must be served for the row to hold), the level, the suite and its version, the date, the provider and host the runs went through, the id they sent, `reads_images`, the pass counts and a one-line English note. The whole list carries `provisional`.
|
||||
|
||||
Today's list has 270 rows. The 11 models of the first ladder results ([08](../proposals/file-agent/08-model-ladder-results.md)), suite `create-and-edit` v1, screening runs of one per cell: Sonnet 5.5, Opus 5.5, Kimi K3, GLM-5.3, DeepSeek V4 Pro and Qwen3.8-27B are `agent`; Haiku 4.5 is `agent_limited`; Qwen3.6-35B-A3B, Gemma 4 31B, Ministral 14B and Qwen3.5-9B are `studio_only`. Then 208 rows from the OpenRouter screening sweep of 7 Oct 2026 ([`tests/live/sweep`](../../surfsense_local/backend/tests/live/sweep/)), suite `openrouter-screen` v1: every OpenRouter model that calls tools, holds 32,768 tokens and costs under $3/M of input, run on the smoke case, then the PDF brief and the board pack, one run each with one re-run of a split; 106 passed both and are `agent`, 102 are `studio_only`. The 18 OpenAI and Grok models an id of 0 had failed were run again once that was fixed. It cost $65.41. And 51 `assumed` rows, never run: models from $3/M of input, the reasoning flagships whose output costs $10/M or more, and the models whose case went past the sweep's $3 stop. 11 models the providers kept rate-limiting have no row and are untested.
|
||||
Today's list has 270 rows. The 11 models of the first ladder results ([08](../proposals/file-agent/08-model-ladder-results.md)), suite `create-and-edit` v1, screening runs of one per cell: Sonnet 5.5, Opus 5.5, Kimi K3, GLM-5.3, DeepSeek V4 Pro and Qwen3.8-27B are `agent`; Haiku 4.5 is `agent_limited`; Qwen3.6-35B-A3B, Gemma 4 31B, Ministral 14B and Qwen3.5-9B are `studio_only`. Then 208 rows from the OpenRouter screening sweep of 7 and 8 Oct 2026 ([`tests/live/sweep`](../../surfsense_local/backend/tests/live/sweep/)), suite `openrouter-screen` v1: every OpenRouter model that calls tools, holds 32,768 tokens and costs under $3/M of input, run on the smoke case, then the PDF brief and the board pack, one run each with one re-run of a split; 106 passed both and are `agent`, 102 are `studio_only`. The 16 OpenAI models an id of 0 had failed, and two Grok models that had passed only on a re-run, were run again once the tools took an optional id of 0 as left out ([agent](agent.md#surfsenses-tools)). The sweep cost $65.41, the re-run $4.68 of it. And 51 `assumed` rows, never run: models from $3/M of input, the reasoning flagships whose output costs $10/M or more, and the models whose case went past the sweep's $3 stop. 11 models the providers kept rate-limiting have no row and are untested.
|
||||
|
||||
Two more suites add rows after the ladder's. `openrouter-screen` v1 is a screening on OpenRouter: a smoke check, then two cases, PDF brief and Board pack. Its row is `agent` when both pass and `studio_only` otherwise, with the counts (`passed` of `counted: 2`) and a note written from them; a pass is `served: remote`, a failure both. `assumed` lists an expensive flagship, $3 or more per million input tokens, at `agent` without runs: `passes` 0 of 0, note "Not run: an expensive flagship assumed to pass", and reason `assumed` rather than counts. The schema refuses an assumed row with runs or below `agent`.
|
||||
Two more suites add rows after the ladder's. `openrouter-screen` v1 is a screening on OpenRouter: a smoke check, then two cases, PDF brief and Board pack. Its row is `agent` when both pass and `studio_only` otherwise, with the counts (`passed` of `counted: 2`) and a note written from them; a pass is `served: remote`, a failure both. `assumed` lists a flagship the sweep did not run, at `agent`: one at $3 or more per million input tokens, one named with the sweep's `--assume`, or one whose case reached the per-case dollar cap ([07-dev-setup](../proposals/file-agent/07-dev-setup.md#live-tests)): `passes` 0 of 0, note "Not run: an expensive flagship assumed to pass", and reason `assumed` rather than counts. The schema refuses an assumed row with runs or below `agent`.
|
||||
|
||||
The list is written, never edited: `python scripts/write_capability_list.py [results.json]` (from `surfsense_local/backend`, with `.` on the path) applies the bar to a ladder results file and rewrites it. The input's shape is in [`ladder_input.py`](../../surfsense_local/backend/scripts/capability_list/ladder_input.py): per model, the id sent, provider, host, where it was served, date, `reads_images`, a note, and one outcome per case (`pass`, `fail`, `fail:<behaviour>` from [08's grouping of failures](../proposals/file-agent/08-model-ladder-results.md#what-the-failures-point-to), `n/a`, `not_run`). The committed input is `capability_list/ladder/create-and-edit-v1.json`, and a test fails when the shipped list's ladder rows are not what the generator writes from it. Rewriting them keeps every other suite's rows that no ladder row names.
|
||||
|
||||
`python scripts/catalog_rows.py sweep-results.json [--date YYYY-MM-DD]` (same place) adds the sweep's rows: each screened model, then each assumed one. The input's shape is in [`sweep_input.py`](../../surfsense_local/backend/scripts/capability_list/sweep_input.py): `models`, each with `id`, `key`, `reads_images`, `status`, `level`, `smoke`, `pdf_brief`, `board_pack`, `passed`, `counted`, `run`, `measured_on`, `cost` and `notes`; `unfinished`; and `assumed`, each with `id`, `key` and `reads_images`. Only a model the sweep has a verdict for (`status` `done`, or no status and every count) gets a row, so a case left unresolved by a provider's errors never reads as a failure; the rest are printed as left out. Its `level` must agree with its counts. A row's date is its own `measured_on`, else the top-level `date`, else `--date`, else today. The list keeps one row per model, by its key or by the looser one, and the stronger row stays, listed or added: the ladder's 8-case rows stand over a screening, and a screening over an assumption, so a later sweep can measure a flagship it assumed. Between rows of one suite the shorter key, the model's own name, wins (gpt-5-2 over gpt-5-2-chat), whichever run came first, and a listed row stays over an added one with the same key; the script prints what each row replaced or why it was left out. A variant whose own model the sweep has no verdict for yet waits for it, rather than stand for it. A case stopped at the per-case dollar cap counts as failed, and the row's note says how many were. The sweep's own `notes` and `cost` stay in its file.
|
||||
`python scripts/catalog_rows.py sweep-results.json [--date YYYY-MM-DD]` (same place) adds the sweep's rows: each screened model, then each assumed one. The input's shape is in [`sweep_input.py`](../../surfsense_local/backend/scripts/capability_list/sweep_input.py): `models`, each with `id`, `key`, `reads_images`, `status`, `level`, `smoke`, `pdf_brief`, `board_pack`, `passed`, `counted`, `run`, `measured_on`, `cost` and `notes`; `unfinished`; and `assumed`, each with `id`, `key` and `reads_images`. Only a model the sweep has a verdict for (`status` `done`, or no status and every count) gets a row, so a case left unresolved by a provider's errors never reads as a failure; the rest are printed as left out. Its `level` must agree with its counts. A row's date is its own `measured_on`, else the top-level `date`, else `--date`, else today. The list keeps one row per model, by its key or by the looser one, and the stronger row stays, listed or added: the ladder's 8-case rows stand over a screening, and a screening over an assumption, so a later sweep can measure a flagship it assumed. Between rows of one suite the shorter key, the model's own name, wins (gpt-5-2 over gpt-5-2-chat), whichever run came first, and a listed row stays over an added one with the same key; the script prints what each row replaced or why it was left out. A variant whose own model the sweep has no verdict for yet waits for it, rather than stand for it. A results file written before the sweep took a capped model as assumed (7 Oct 2026) counted a capped case as failed, and such a row's note says how many were. The sweep's own `notes` and `cost` stay in its file. How to run the sweep and then this script, and how to take a re-run's verdict for a model the list already screened, is in [07-dev-setup](../proposals/file-agent/07-dev-setup.md#live-tests).
|
||||
|
||||
## Matching
|
||||
|
||||
|
|
@ -57,15 +57,16 @@ A chat is Basic (Q&A), the chat engine's retrieval answers with Studio beside it
|
|||
|
||||
- `SelectionRead.capability` on `GET` and `PUT /llm/selection/text_gen` (null for other slots): `level`, `label_key` (the ICU select branch), `reason` (`code`, `values`), `note`, `measured` (`key`, `suite`, `assumed`, `suite_version`, `measured_on`, `provider`, `host`, `reads_images`, `passed`, `counted`, `provisional`, or null) and `modes` (`agentic_allowed`, `blocked`, `default_mode`: `basic` or `agentic`, `reason` (`code`, `values`), `remembered_mode`).
|
||||
- `capability_level` on each row of `GET /llm/connections/{id}/models` and `GET /llm/providers/llamacpp/models` that can fill the chat slot.
|
||||
- `POST /workspaces/{id}/chat/threads` takes `mode` ([chat](chat.md#agent-threads)).
|
||||
- `POST /workspaces/{id}/chat/threads` takes `mode` and `remember` ([chat](chat.md#agent-threads)).
|
||||
|
||||
In dev builds only, the model picker tags each chat model by how it did in the Agentic tests: "Agentic", "Agentic, may need nudges", "Low Agentic score" or "Not tested"; production shows no tag. Settings › Models shows the chat model's level, its evidence line and its note, or for an assumed row only "Not run: SurfSense expects this expensive flagship model to pass."; then the mode new chats start in, and either that the mode is chosen per chat in the composer or why Agentic is not available.
|
||||
|
||||
The composer's switch ([`mode-picker.tsx`](../../surfsense_local/frontend/src/features/chat/modes/mode-picker.tsx)) is a menu of "Basic (Q&A)" and "Agentic", each with one line on what it does. On a new chat it starts at the model's `default_mode` and follows the model when it changes; a pick holds for that model for the session, in a context above the open workspace. The new thread's `mode` is always sent, with `remember: true` only when the user picked it, so only a pick becomes the model's default. On a new chat each mode gets a short line, and an Agentic that did not pass carries a tag ("Not tested", "Low score", "May need nudges", "Local copy", "Expected to pass"); the `reason`, or a gate, shows in a tooltip beside the Agentic row on hover or focus and is read with the row, in both menus: "Passed 2 of 8 Agentic tests, so it may stop early or make mistakes.", "Not tested with this model yet, so it may stop early or make mistakes.", "Passed on its full-size version. A local copy may do worse.", or that a flagship is assumed. Nothing is said under the composer. A gated Agentic is disabled with its reason, such as "This model can’t use tools." In an open chat, under "This chat’s mode", the switch lists the same two rows with the chat's mode checked; the other is marked "New chat", and picking it picks that mode and opens a new chat. A refused Agentic chat is worded by its code in a toast, and the question and images go back into the composer. A refusal by a gate the selection could not say, such as a local model's tool calls, disables Agentic with its reason for that model for the session ([`new-chat-mode.ts`](../../surfsense_local/frontend/src/features/chat/modes/new-chat-mode.ts)), so the next chat starts in Basic (Q&A) rather than asking again. The agent reads no attached image yet, so an Agentic chat, new or open, takes none ([chat](chat.md)).
|
||||
The composer's switch ([`mode-picker.tsx`](../../surfsense_local/frontend/src/features/chat/modes/mode-picker.tsx)) is a menu of "Basic (Q&A)" and "Agentic", each with one line on what it does. On a new chat it starts at the model's `default_mode` and follows the model when it changes; a pick holds for that model for the session, in a context above the open workspace. The new thread's `mode` is always sent, with `remember: true` only when the user picked it, so only a pick becomes the model's default. An Agentic that did not pass carries a tag ("Not tested", "Low score", "May need nudges", "Local copy", "Expected to pass"); the `reason`, or a gate, shows in a tooltip beside the Agentic row on hover or focus and is read with the row, in both menus: "Passed 2 of 8 Agentic tests, so it may stop early or make mistakes.", "Not tested with this model yet, so it may stop early or make mistakes.", "Passed on its full-size version. A local copy may do worse.", or that a flagship is assumed. Nothing is said under the composer. A gated Agentic is disabled with its reason, such as "This model can’t use tools." In an open chat, under "This chat’s mode", the switch lists the same two rows with the chat's mode checked; the other is marked "New chat", and picking it picks that mode and opens a new chat. A refused Agentic chat is worded by its code in a toast, and the question and images go back into the composer. A refusal by a gate the selection could not say, such as a local model's tool calls, disables Agentic with its reason for that model for the session ([`new-chat-mode.ts`](../../surfsense_local/frontend/src/features/chat/modes/new-chat-mode.ts)), so the next chat starts in Basic (Q&A) rather than asking again. The agent reads no attached image yet, so an Agentic chat, new or open, takes none ([chat](chat.md)).
|
||||
|
||||
## Known gaps
|
||||
|
||||
- Every row is screening (n=1) on OpenRouter's default routing or Anthropic's compatibility layer; 05's committed rows, three runs per cell with the provider order pinned, replace it.
|
||||
- No row is measured on this computer, so every local model is `not_measured`; llama.cpp runs need a `served: local` row.
|
||||
- The matrix runner writes per-run `result.json` files, not the ladder input; each column is mapped by hand, triage labels included.
|
||||
- The sweep's rows come from a committed input, [`capability_list/sweep/openrouter-screen-v1.json`](../../surfsense_local/backend/scripts/capability_list/sweep/openrouter-screen-v1.json): the screen of 7 and 8 Oct 2026 with the 18 re-run models' verdicts in place of their first ones. Its two runs are beside it under `sweep/runs/`, each a `sweep-results.json` and its `attempts.jsonl`; their run folders, about 840 MB of transcripts and files, stay out of the repository. `catalog_rows.py` cannot fold a re-run into a screen, so that input was put together by hand, and no test pins the sweep's rows to it as one pins the ladder's ([#2221](https://github.com/MODSetter/SurfSense/issues/2221)). The 11 models the sweep left unresolved have no row ([#2222](https://github.com/MODSetter/SurfSense/issues/2222)).
|
||||
- The row's note is English; the interface shows it untranslated.
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ worker-studio ── HTTP ──> sd-server, audiocpp, remote image and speech e
|
|||
worker-ingest, worker-studio ── POST /internal/events ──> api ── SSE /workspaces/{id}/events
|
||||
```
|
||||
|
||||
- Electron's main process starts four sidecars at boot and supervises them ([`index.ts`](../../surfsense_local/electron/src/main/index.ts), [`sidecars/`](../../surfsense_local/electron/src/main/sidecars/)): the API, one Huey worker per queue, and llama-server. Packaged builds run frozen binaries from the app's resources; `pnpm dev` runs `main.py` and `worker.py <queue>` on the backend's `.venv` interpreter, synced by `predev`'s own `uv run` scripts. Not through `uv run`: Windows kills Electron's children with it but not theirs, so after a Ctrl-C the last session's Studio worker, still reading the queue, took the next jobs.
|
||||
- Electron's main process starts four sidecars at boot and supervises them ([`sidecars/boot.ts`](../../surfsense_local/electron/src/main/sidecars/boot.ts), [`sidecars/`](../../surfsense_local/electron/src/main/sidecars/)): the API, one Huey worker per queue, and llama-server. Packaged builds run frozen binaries from the app's resources; `pnpm dev` runs `main.py` and `worker.py <queue>` on the backend's `.venv` interpreter, synced by `predev`'s own `uv run` scripts. Not through `uv run`: Windows kills Electron's children with it but not theirs, so after a Ctrl-C the last session's Studio worker, still reading the queue, took the next jobs.
|
||||
- llama-server starts whenever its pinned build is staged, in dev too (`pnpm build:llamacpp`, which `predev` runs). It reads per-model arguments from a preset file once at startup, so Electron restarts it when the API rewrites that file ([`local-models/runtime.md`](local-models/runtime.md)).
|
||||
- sd-server takes its model as startup arguments, so it cannot start at boot. Whenever its pinned build is staged, in dev too (`pnpm build:sdcpp`, which `predev` runs), `watchImageModel` asks the API every 5 seconds which files to run, and starts, restarts or stops it on a change. The API names a model only while a Studio job needs one and for up to 5 minutes after, so its weights are not held beside the chat model all session ([`studio.md`](studio.md#the-image-path)).
|
||||
- audiocpp_server refuses an empty model list, so it starts only once the API has written `audio/server.json` under the data directory, in dev too (`pnpm build:audiocpp`, which `predev` runs). On Windows and Linux that script compiles audio.cpp, and without a C++ toolchain the app runs without local audio ([packaging](packaging.md)). Electron checks that file every 5 seconds and restarts the server when it changes. Electron sets the machine-wide flags: the CPU backend, half the logical cores up to 8, one loaded model, and an unload after 5 idle minutes. The API writes that file whenever an audio model is installed or deleted, and at startup ([`local-models/catalog.md`](local-models/catalog.md)). The Studio worker voices podcasts there, at `SURFSENSE_LOCAL_AUDIO_BASE_URL`, and unloads the model when a podcast ends ([`studio.md`](studio.md)).
|
||||
|
|
@ -101,7 +101,7 @@ The rules that keep the processes out of each other's way:
|
|||
- The preload bridge is the renderer's only other channel: opening or revealing an original file natively, opening an external link, the platform name, updates, theme, the title bar, this session's log and unexpected sidecar-exit notices. The renderer never sees Node or the sidecars directly. A source with a registered viewer (PDF first) is fetched from the API and previewed in place of the left sidebar; other file types keep the native open action ([`documents.md`](documents.md)).
|
||||
- API calls go through `request()` in [`lib/api.ts`](../../surfsense_local/frontend/src/lib/api.ts), except the Studio viewers, which fetch an artifact's file bytes directly; an `<img>`, `<audio>` or download link takes its absolute address from `apiUrl()`. `request()` prefixes the address, turns an error body into an `ApiError` with its `code`, and when a user action is refused with `403 egress_disabled` it asks for consent and retries once.
|
||||
- TanStack Query holds most server state: threads and messages, the citation panel, the model catalog and settings, the Studio formats and artifact list, and an open artifact with what its viewer reads. An `artifacts` event invalidates the list and every open artifact under one key prefix ([`features/studio/query-keys.ts`](../../surfsense_local/frontend/src/features/studio/query-keys.ts)), so a regenerated artifact that is open shows its new generation. The sources list is component state, reloaded on the events above.
|
||||
- There is no router. [`app/app-bootstrap.tsx`](../../surfsense_local/frontend/src/app/app-bootstrap.tsx) asks `GET /llm/onboarding`, then renders either onboarding ([`local-models/selection.md`](local-models/selection.md)) or the lazily loaded dashboard: the workspace rail, threads and sources on the left, the conversation in the middle, and Studio, artifacts and the citation panel on the right.
|
||||
- There is no router. [`app/app-bootstrap.tsx`](../../surfsense_local/frontend/src/app/app-bootstrap.tsx) asks `GET /llm/onboarding`, then renders either onboarding ([`local-models/selection.md`](local-models/selection.md)) or the lazily loaded dashboard: the workspace rail, threads and sources on the left, the conversation in the middle, and Studio, artifacts and the citation panel on the right. The left and right columns resize by dragging their inner edge, and the conversation keeps at least 520 px ([studio](studio.md#the-studio-panel)).
|
||||
- Code is grouped by feature under `src/features/`. Each HTTP contract lives in one feature's `api.ts`, next to its hooks and components, and other features import it from there.
|
||||
|
||||
## Codemap
|
||||
|
|
@ -110,9 +110,9 @@ The rules that keep the processes out of each other's way:
|
|||
- [`backend/modules/`](../../surfsense_local/backend/modules/): one folder per feature, holding whichever of its models, schemas, router and service it needs: `workspaces`, `documents`, `chunks`, `chat`, `artifacts` (Studio), `events`, `llm` (runtimes, catalog, connections, selection), `egress`, `agent`, `license`, `migration` (import) and `health`. `shared.db.import_models()` imports every model at startup, because relationships name their targets as strings.
|
||||
- [`backend/worker/`](../../surfsense_local/backend/worker/): `consumer.py` drains one queue per process, `jobs.py` holds status transitions and cancellation, `notify.py` the change notice, and `ingestion/` and `studio/` the two pipelines.
|
||||
- [`backend/shared/`](../../surfsense_local/backend/shared/): configuration, the engine and its pragmas, `upgrade_to_head()`, the two Huey queues, `retrieve()`, and the secret that encrypts stored keys.
|
||||
- [`backend/alembic/`](../../surfsense_local/backend/alembic/): revisions `0001` to `0012`, all hand-written. `env.py` has no `target_metadata`, so autogenerate cannot run by accident.
|
||||
- [`backend/alembic/`](../../surfsense_local/backend/alembic/): revisions `0001` to `0027`, all hand-written. `env.py` has no `target_metadata`, so autogenerate cannot run by accident.
|
||||
- [`frontend/src/features/`](../../surfsense_local/frontend/src/features/): `chat`, `agent`, `sources`, `studio`, `workspaces`, `dashboard`, `models`, `onboarding`, `settings`, `egress`, `license`, `migration`, `updates` and `feedback`.
|
||||
- [`electron/src/main/`](../../surfsense_local/electron/src/main/): `index.ts` (boot, window, IPC and the image-model, preset, audio and agent watchers), `sidecars/` (the supervisor and one spec per sidecar), `session-log/` (this run's output, for issue reports), `secret.ts`, `updater.ts` and `document-files.ts`.
|
||||
- [`electron/src/main/`](../../surfsense_local/electron/src/main/): `index.ts` (the window, IPC and the paths it hands to boot), `sidecars/` (boot, the supervisor, one spec per sidecar, and the image-model, preset, audio and agent watchers), `session-log/` (this run's output, for issue reports), `secret.ts`, `updater.ts` and `document-files.ts`.
|
||||
|
||||
## Where to read next
|
||||
|
||||
|
|
@ -127,11 +127,13 @@ The rules that keep the processes out of each other's way:
|
|||
- [Connections](connections.md): OpenAI-compatible endpoints and where keys live.
|
||||
- [ChatGPT subscription](chatgpt-subscription.md): a connection signed in with a ChatGPT account, answered through the Responses API.
|
||||
- Local models: [runtime](local-models/runtime.md), [fit](local-models/fit.md), [admission and parallel slots](local-models/admission.md), [catalog](local-models/catalog.md), [selection and onboarding](local-models/selection.md).
|
||||
- [Localization](localization.md): the interface in English, Japanese and German.
|
||||
- [Localization](localization.md): the interface in ten languages.
|
||||
- [Egress](egress.md), [Office pack](office-pack.md), [import](import.md), [license in the app](license/app.md), [license portal](license/portal.md), [updates](updates.md), [about](about.md), [issue reports](issue-reports.md), [resource usage](resource-usage.md), [packaging](packaging.md), [sunset](sunset.md).
|
||||
- [Contracts](../contracts/README.md) between the trees.
|
||||
|
||||
## Known gaps
|
||||
|
||||
- The sources list reloads itself on a workspace event rather than through TanStack Query: it is component state, so nothing else that shows a document is invalidated by the same event.
|
||||
- When the API does not start, because it exits or `/health` does not answer within 60 seconds, Electron writes the reason to the session log and quits: no window opens, nothing on screen says why, and the log, kept only in memory, goes with the app ([about](about.md#known-gaps)). A migration snapshot that cannot be written, such as on a full disk, ends this way ([data model](data-model.md#known-gaps)).
|
||||
- The supervisor listens for no `error` event on the children it spawns ([`supervisor.ts`](../../surfsense_local/electron/src/main/sidecars/supervisor.ts)), so an executable that cannot be spawned at all throws in the main process instead of being reported as a crash ([#2228](https://github.com/MODSetter/SurfSense/issues/2228)).
|
||||
|
||||
|
|
|
|||
|
|
@ -128,13 +128,13 @@ The Linux runner is pinned because `ubuntu-latest` moves to 26.04 and would sile
|
|||
|
||||
Each runner, in order, checks the version (semver; on a tag push it must equal `surfsense_local/VERSION`; whenever it publishes it must carry no prerelease suffix), refuses to build with the test signing key ([license](license/app.md)), freezes the binaries, smokes the frozen worker's Docling vision imports (`--check-vision-runtime`) and the frozen API's `/health`, stages the model packs, builds the SPA and the Electron bundles, stages llama.cpp, opencode, audio.cpp and sd.cpp, and runs `electron-builder`. On macOS and Windows, electron-builder's `afterSign` hook, `scripts/opencode/after-sign.mjs`, runs `check-stage.mjs` on the packaged opencode folder once the app is signed, and on macOS notarized, before any installer is made, so a tag's build never uploads one whose opencode signing broke; unsigned builds run it too. Linux signs nothing, so electron-builder skips the hook there with a warning. Every runner then runs `check-stage.mjs` again on the folder its build packaged: `win-unpacked`, `linux-unpacked` or `mac-arm64/SurfSense.app`. The three runners wait for the `audiocpp` and `sdcpp` jobs, and unpack audio.cpp's Windows and Linux builds and sd.cpp's Linux and macOS builds. On Linux it then starts the API from the packaged `linux-unpacked` resources and runs `llama-server --list-devices`, `audiocpp_server --list-devices` and `sd-server --help` from their packaged directories, because ggml finds its backends only next to the running executable, and checks that eSpeak-ng and its licence are packaged beside `audiocpp_server`, and sd.cpp's Vulkan backend and licence beside `sd-server`.
|
||||
|
||||
The macOS build is signed with the hardened runtime and notarized whenever the signing secrets are available. electron-builder signs every Mach-O file in the bundle, `Contents/Resources` included, with `entitlementsInherit`, except `opencode`: `mac.signIgnore` leaves it with its vendor's Developer ID signature, hardened runtime, secure timestamp and the five entitlements Bun asks for, among them `disable-executable-page-protection`, which the app's file lacks (**measured** on the pinned `opencode-darwin-arm64.zip`, 7 Oct 2026). `rg` arrives only ad-hoc signed, so it is signed with the app. Two steps run only on tag pushes: the early check of the Apple notarization credentials, and Azure Trusted Signing for the Windows build, which signs every `.exe` under `resources/` as electron-builder copies it, replacing any signature it carries. `opencode.exe` arrives Authenticode-signed by its vendor and `rg.exe` unsigned, so a dry run runs those, and a tag's build runs both signed with SurfSense's certificate.
|
||||
The macOS build is signed with the hardened runtime and notarized whenever the signing secrets are available, a dry run in this repository included. electron-builder signs every Mach-O file in the bundle, `Contents/Resources` included, with `entitlementsInherit`, except `opencode`: `mac.signIgnore` leaves it with its vendor's Developer ID signature, hardened runtime, secure timestamp and the five entitlements Bun asks for, among them `disable-executable-page-protection`, which the app's file lacks (**measured** on the pinned `opencode-darwin-arm64.zip`, 7 Oct 2026). The notary service takes it so: v2.2.0's tag build was notarized carrying it (9 Oct 2026). `rg` arrives only ad-hoc signed, so it is signed with the app. Two steps run only on tag pushes: the early check of the Apple notarization credentials, and Azure Trusted Signing for the Windows build, which signs every `.exe` under `resources/` as electron-builder copies it, replacing any signature it carries. `opencode.exe` arrives Authenticode-signed by its vendor and `rg.exe` unsigned, and a tag's build runs both signed with SurfSense's certificate. A dry run has no Azure credentials, but `CSC_LINK`, the macOS certificate, reaches the Windows runner too, so electron-builder signs every `.exe` in the app with it through signtool (the 2.1.0 dry run's log, 3 Oct 2026); opencode's and rg's would be signed the same way, though no dry run has staged them yet.
|
||||
|
||||
The installer's version is the one the workflow resolved, passed as `-c.extraMetadata.version`. The tree carries it too: `surfsense_local/VERSION` is the one number, and `surfsense_local/scripts/bump-version.sh` writes it into the backend's `pyproject.toml`, both `package.json` files, the site's `lib/app-release.ts` and the backend's `app/license/release.py`, then refreshes `uv.lock`.
|
||||
|
||||
## The NSIS size limit
|
||||
|
||||
`makensis` is 32-bit and cannot map a payload over 2 GB, so the Windows installer has a hard ceiling. The Windows build hit it before 2.0.0 through the bundled Ollama's CUDA runners, which were pruned; 2.0.2's `SurfSense-Setup.exe` is 1,296,981,808 bytes on the release, about 1.3 GB. No CUDA payload ships.
|
||||
`makensis` is 32-bit and cannot map a payload over 2 GB, so the Windows installer has a hard ceiling. The Windows build hit it before 2.0.0 through the bundled Ollama's CUDA runners, which were pruned; 2.2.0's `SurfSense-Setup.exe`, the first to carry opencode and ripgrep, is 1,209,002,840 bytes on the release, about 1.2 GB. No CUDA payload ships.
|
||||
|
||||
## Packaging tests
|
||||
|
||||
|
|
@ -156,5 +156,8 @@ Three run in CI: `test_license_key.py` inside the release workflow, `test_audioc
|
|||
|
||||
- No issue on audio.cpp asks for archives that meet the app's floors yet, so `build-audiocpp.yml` has no end date.
|
||||
- No packaging test checks that the frozen API carries the agent's `SKILL.md`; `api.spec` collects it, and it was checked by hand.
|
||||
- No notarized build has carried opencode yet, so it is unproven that the notary service takes it under its vendor's team ID; it should, since it checks each binary for a Developer ID signature, the hardened runtime and a timestamp, not for the app's team.
|
||||
- `test_frozen_document_scripts.py` runs in no workflow, so nothing in CI checks that a packaged worker can run a document or analysis script ([#2227](https://github.com/MODSetter/SurfSense/issues/2227)).
|
||||
- [`desktop-tests.yml`](../../.github/workflows/desktop-tests.yml) runs on pull requests and by hand (`workflow_dispatch`), never on a push to `dev`, and only on Linux, so a break that lands on `dev`, or shows only on Windows or macOS, shows first in a release build ([#2226](https://github.com/MODSetter/SurfSense/issues/2226)).
|
||||
- The packaged `llama-server`, `audiocpp_server` and `sd-server` are started only on the Linux runner ([#1986](https://github.com/MODSetter/SurfSense/issues/1986)).
|
||||
- A dry run's Windows build is signed with the macOS certificate through signtool, not through Azure Trusted Signing, so only a tag's build exercises Windows signing as it ships, and a lapsed Azure client secret shows there first ([#2223](https://github.com/MODSetter/SurfSense/issues/2223)).
|
||||
- What matplotlib adds to the installer has not been measured against its size gate. A scratch freeze of the worker put it at about 15 MB unpacked under `_internal/matplotlib`, plus about 0.6 MB for contourpy and kiwisolver.
|
||||
|
|
|
|||
|
|
@ -279,7 +279,7 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i
|
|||
| Format | Viewer |
|
||||
|---|---|
|
||||
| `summary` | the markdown, rendered with Streamdown |
|
||||
| `docx` | rendered in the app with docx-preview, inside a frame whose policy loads nothing but the file's own data-URL images and fonts and runs no script; no altChunk is rendered, and a link to anywhere but a place in the file is dropped; tracked changes show, insertions underlined and deletions struck through; comments show as Word shows them, each passage shaded with a numbered marker and its comment in a balloon in the margin level with it ([`docx-comments.ts`](../../surfsense_local/frontend/src/features/studio/viewers/docx-comments.ts)) |
|
||||
| `docx` | rendered in the app with docx-preview, inside a frame whose policy loads nothing but the file's own data-URL images and fonts and runs no script; no altChunk is rendered, and a link to anywhere but a place in the file is dropped; tracked changes show, insertions underlined and deletions struck through; comments show as Word shows them, each passage shaded with a numbered marker and its comment in a balloon in the margin level with it, pushed below the one above, and hovering a balloon deepens its passage's shade ([`docx-comments.ts`](../../surfsense_local/frontend/src/features/studio/viewers/docx-comments.ts)); the page opens fitted to the frame's width, never above 100%, with room beside it for the balloons when there are comments, and zooms from 25% to 300% with its buttons or Ctrl and the wheel |
|
||||
| `pptx` | rendered in the app with `@aiden0z/pptx-renderer` |
|
||||
| `xlsx` | parsed with ExcelJS |
|
||||
| `pdf` | pdf.js |
|
||||
|
|
@ -297,9 +297,10 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i
|
|||
## The Studio panel
|
||||
|
||||
- Studio lives in the right rail: pick a format, pick sources, add an optional prompt, generate. The source picker is the same included set as the sources panel, so chat and Studio share one selection. An unavailable format shows the interface's sentence for the API's `unavailable_code`, or the API's `unavailable_reason` for a code it does not know. A create or regenerate refused for the same reason shows it the same way in the panel's alert.
|
||||
- The rail and the left sidebar resize by dragging their inner edge, or with the arrow keys on a focused edge; a double-click on an edge restores its default. The sidebar goes from 272 to 560 px and the rail from 400 to 640; a source preview or an inspected citation or artifact goes up to 800. Neither takes the chat below 520 px, and on a narrow window the sidebar gives way first. The widths are kept in local storage ([`column-widths.ts`](../../surfsense_local/frontend/src/features/dashboard/column-widths.ts)). They live in the section that holds the columns, so a drag renders the column boxes and their edges, never the sidebar, the thread or Studio inside them ([`dashboard-columns.tsx`](../../surfsense_local/frontend/src/features/dashboard/dashboard-columns.tsx)).
|
||||
- The rail and the left sidebar resize by dragging their inner edge, or from the keyboard on a focused edge, a `separator` named "Resize sidebar" or "Resize right panel" that carries its width: an arrow moves it 16 px, and Home and End take it to its limits ([`resize-handle.tsx`](../../surfsense_local/frontend/src/components/ui/resize-handle.tsx)). A double-click on an edge restores its default, and a collapsed rail has no edge. The sidebar goes from 272 to 560 px and the rail from 400 to 640; a source preview or an inspected citation or artifact goes up to 800. Neither takes the chat below 520 px, and on a narrow window the sidebar gives way first. The widths are kept in local storage ([`column-widths.ts`](../../surfsense_local/frontend/src/features/dashboard/column-widths.ts)). They live in the section that holds the columns, so a drag renders the column boxes and their edges, never the sidebar, the thread or Studio inside them ([`dashboard-columns.tsx`](../../surfsense_local/frontend/src/features/dashboard/dashboard-columns.tsx)).
|
||||
- A podcast waits for its brief: the panel loads `GET .../studio/podcast/brief` and renders a form for style, duration and speakers before the job can be submitted.
|
||||
- The artifact list shows each artifact with its status, and each row can be opened, regenerated, cancelled or deleted. A script document's versions share one row ([above](#script-documents)).
|
||||
- A name its column cuts off, an artifact's title in the list, a source's in the panel's source picker, or the title in the header of an open artifact, citation or source preview, shows whole in a tooltip on hover or keyboard focus, and only while it is cut off ([`overflow-tooltip.tsx`](../../surfsense_local/frontend/src/components/ui/overflow-tooltip.tsx)).
|
||||
|
||||
## The image path
|
||||
|
||||
|
|
@ -323,3 +324,4 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i
|
|||
- A podcast is WAV. The design encodes MP3 with a bundled ffmpeg, which is not built.
|
||||
- A revised copy is cut from its design ([03](../proposals/file-agent/03-editable-artifacts.md)): versions are artifacts, not `artifact_versions` rows, and engines run in the Studio worker's thread, not an engines queue's child process; the agent anchors edits by quote, with no `inspect_document` ids; there are no partial workflows, no per-change accept or reject and no side list of changes, no "With changes (internal)" download and no leak scan, no review record, no reconcile when the original changes, and no "Make a revised copy" in the source viewer. Word headers, footers and footnotes are not edited. An edited workbook is not recalculated until Excel opens it, and a deck has no thumbnails or diff. The report's codes reach the user in English through the agent: the interface does not render them.
|
||||
- The Word viewer draws tracked changes and comments with docx-preview's experimental rendering. Its 0.4.0 drops what a fragment holds, which is how it makes a comment's reference, and throws on a comment with no author, so the viewer passes it an element maker that keeps the one and fills the other; a file whose comments it still cannot lay out shows without them. A comment's time shows as Word shows it, unconverted, since Word writes its own clock's time marked UTC. The balloons take about 230 px beside the page, so a file with comments fits the frame at a smaller zoom.
|
||||
- The Word viewer fits a file to the frame once, when it loads: dragging the rail's edge afterwards keeps that zoom, where the PDF viewer refits to the page width until it is zoomed by hand ([#2219](https://github.com/MODSetter/SurfSense/issues/2219)).
|
||||
|
|
|
|||
|
|
@ -55,3 +55,4 @@ Artifact names carry no version, so the root README's links to `/releases/latest
|
|||
## Known gaps
|
||||
|
||||
- The deb build likely auto-updates, although the launch plan said it would not: electron-builder writes `resources/package-type` for a deb, electron-updater picks its deb updater from that file, and v2.0.2's `stable-linux.yml` on the release lists `SurfSense.deb`. Not checked on an installed deb.
|
||||
- Nothing takes an installed previous version through an update before users get it. The updater reads only published releases, so [`RELEASE.md`](../../surfsense_local/RELEASE.md)'s check from the previous version is by hand: the draft's installer over that version before publishing, the update itself only after.
|
||||
|
|
|
|||
|
|
@ -19,9 +19,9 @@ Source links below are to opencode at tag [`v1.18.34`](https://github.com/anomal
|
|||
- On x64, stage the `-baseline` archive, which opencode builds without AVX2 (`avx2: false` in [`opencode/script/build.ts`](https://github.com/anomalyco/opencode/blob/v1.18.34/packages/opencode/script/build.ts)), so the agent runs on older CPUs too.
|
||||
- Ship ripgrep beside it and put that folder on the `PATH` opencode gets. On its first grep, opencode looks for `rg` on `PATH`, then in its own `bin` folder under its cache directory, and otherwise downloads it from GitHub ([`core/src/ripgrep/binary.ts`](https://github.com/anomalyco/opencode/blob/v1.18.34/packages/core/src/ripgrep/binary.ts)).
|
||||
- Ship opencode's one config folder with `@opencode-ai/plugin` in `node_modules` and a `package-lock.json` that lists it. At startup opencode installs that package into every config folder in the background, unless it is already there ([`opencode/src/config/config.ts`](https://github.com/anomalyco/opencode/blob/v1.18.34/packages/opencode/src/config/config.ts) L452, [`core/src/npm.ts`](https://github.com/anomalyco/opencode/blob/v1.18.34/packages/core/src/npm.ts)). Launch it with `npm_config_audit=false` and `npm_config_fetch_retries=0`, so a case this misses fails at once instead of waiting on a network it cannot reach.
|
||||
- On macOS the executable keeps the signature it arrives with: Developer ID, hardened runtime and the entitlements Bun's engine uses ([Bun docs](https://bun.com/docs/guides/runtime/codesign-macos-executable)), one of which, `disable-executable-page-protection`, the app's [`entitlements.mac.plist`](../../../surfsense_local/electron/build/entitlements.mac.plist) lacks; `mac.signIgnore` keeps electron-builder from replacing it ([packaging](../../architecture/packaging.md)). Confirm a notarized build runs it, and that the signed Windows build runs it on a clean machine with Defender on.
|
||||
- On macOS the executable keeps the signature it arrives with: Developer ID, hardened runtime and the entitlements Bun's engine uses ([Bun docs](https://bun.com/docs/guides/runtime/codesign-macos-executable)), one of which, `disable-executable-page-protection`, the app's [`entitlements.mac.plist`](../../../surfsense_local/electron/build/entitlements.mac.plist) lacks; `mac.signIgnore` keeps electron-builder from replacing it ([packaging](../../architecture/packaging.md)). v2.2.0's release CI ran it from the notarized app. Confirm that the signed Windows build runs it on a clean machine with Defender on ([#2230](https://github.com/MODSetter/SurfSense/issues/2230)).
|
||||
- The installer downloads nothing. Today's targets are already offline builds: NSIS on Windows, dmg and zip on macOS, AppImage and deb on Linux ([`electron-builder.yml`](../../../surfsense_local/electron/electron-builder.yml)).
|
||||
- The 2.0.3 installers are 1.13 GB on Windows to 1.36 GB for the AppImage ([release](https://github.com/MODSetter/SurfSense/releases/tag/v2.0.3)), against a 2 GB `makensis` ceiling on Windows ([packaging](../../architecture/packaging.md)). They grow by about the compressed archive, 46 to 62 MB. Measure them after adding opencode and ripgrep.
|
||||
- The 2.0.3 installers are 1.13 GB on Windows to 1.36 GB for the AppImage ([release](https://github.com/MODSetter/SurfSense/releases/tag/v2.0.3)), against a 2 GB `makensis` ceiling on Windows ([packaging](../../architecture/packaging.md)). They grow by about the compressed archive, 46 to 62 MB. v2.2.0, the first release to carry opencode and ripgrep, is 1.21 GB on Windows to 1.45 GB for the AppImage, 79 to 92 MB more than 2.1.0 with that release's other changes ([release](https://github.com/MODSetter/SurfSense/releases/tag/v2.2.0)).
|
||||
|
||||
## Launch
|
||||
|
||||
|
|
|
|||
|
|
@ -34,6 +34,8 @@ opencode came first, for a model known to call tools ([`01-which-engine.md`](01-
|
|||
|
||||
## Locked decisions
|
||||
|
||||
The [file agent proposal](../file-agent/README.md#locked-decisions-of-the-agent-proposal-this-changes) changes several of these, and what shipped follows it: since 7 Oct 2026 the user picks Basic (Q&A) or Agentic for each new chat, and every model may pick Agentic unless a technical gate keeps it out ([`01-which-engine.md`](01-which-engine.md#decision)); the agent has no shell ([ADR 0039](../../adr/0039-document-scripts-run-without-approval.md)); and it edits the user's files only into new copies, never the file itself ([agent](../../architecture/agent.md#revised-copies-the-agent-makes)).
|
||||
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| Engines | Two. opencode for a model on a tested list; fixed workflows for every other model. |
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
---
|
||||
status: proposed
|
||||
status: in-progress
|
||||
tracking: https://github.com/MODSetter/SurfSense/issues/2100
|
||||
code:
|
||||
- docker/local/
|
||||
- surfsense_local/electron/src/main/sidecars/
|
||||
|
|
@ -16,6 +17,8 @@ code:
|
|||
|
||||
> The desktop app's stack, minus Electron, as one container: a supervisor shared with Electron runs the API, the workers and the model servers on loopback, and Caddy is the only thing listening. It replaces the legacy self-host compose. The decision is [ADR 0035](../../adr/0035-docker-compose-runs-the-desktop-stack.md); GPU images are in [`gpu.md`](gpu.md).
|
||||
|
||||
**Status, 11 Oct 2026:** [steps](#steps) 1 to 3 are built: the CORS origins setting ([#2122](https://github.com/MODSetter/SurfSense/pull/2122)), the `VITE_API_BASE` fallback ([#2134](https://github.com/MODSetter/SurfSense/pull/2134)), and boot and the watchers in `sidecars/` with no Electron import ([#2215](https://github.com/MODSetter/SurfSense/pull/2215)).
|
||||
|
||||
## What changes
|
||||
|
||||
| | Legacy self-host | With this |
|
||||
|
|
@ -50,7 +53,7 @@ One container, because the stack assumes one machine: SQLite and the Huey queue
|
|||
|
||||
## The supervisor, shared with Electron
|
||||
|
||||
The sidecar specs and the process supervisor in [`electron/src/main/sidecars/`](../../../surfsense_local/electron/src/main/sidecars/) already import only Node. What ties boot to Electron is in [`index.ts`](../../../surfsense_local/electron/src/main/index.ts): `bootSidecars()` reads `app.isPackaged` and `process.resourcesPath`, and the three watchers live beside it.
|
||||
The sidecar specs and the process supervisor in [`electron/src/main/sidecars/`](../../../surfsense_local/electron/src/main/sidecars/) already import only Node. Since [#2215](https://github.com/MODSetter/SurfSense/pull/2215), boot ([`boot.ts`](../../../surfsense_local/electron/src/main/sidecars/boot.ts)) and the watchers live in `sidecars/` and take their paths as a context, which [`index.ts`](../../../surfsense_local/electron/src/main/index.ts) builds from `app.isPackaged` and `process.resourcesPath`.
|
||||
|
||||
- Boot and the watchers move into `sidecars/`, taking their paths as a context instead of reading Electron. The watchers restart llama-server when `models/models.ini` changes, restart audiocpp when `audio/server.json` changes, and start or stop sd-server from `/llm/image/local/runtime`.
|
||||
- `electron/src/headless/` is the container's entry. It builds that context from the environment, runs the start-up checks, boots, waits for `/health`, then starts Caddy as one more sidecar. It forwards SIGTERM to every child. Node 24 runs it as TypeScript, as the package's tests already run, so it has no build step.
|
||||
|
|
|
|||
|
|
@ -30,6 +30,8 @@ code:
|
|||
|
||||
> A workspace's sources sit in folders. Every workspace has one library root whose folders the user makes in the app. It may also link folders on disk as further roots. SurfSense indexes a linked folder in place, watches it, and never writes to it. A delete in the library goes to a Trash first, and a linked folder that seems to lose most of its files asks before anything is removed. What the user ticks in the tree is a source scope of folder and document ids. The scope is stored on the chat thread, recorded on each turn and artifact version, and resolved by the server, so chat, Studio and the agent read the same sources however many there are. The agent sees the scope as a per-thread folder of text files that mirrors the tree. Engines reach an original only by its source id. The first phase needs no folders: it fixes the bug that hides every source after the 50th from chat and Studio, and it snapshots the database before any migration runs.
|
||||
|
||||
**Status, 11 Oct 2026:** Library folders, adding a whole folder, the server-side scope with tri-state ticks, and per-chat agent folders (phases 3a and 3b) are built, and 2.2.0 is the first release with them ([documents](../../architecture/documents.md#folders), [agent](../../architecture/agent.md#folders)). The Trash and linking a folder on disk (phases 4a and 4b) are designed, not built.
|
||||
|
||||
This is one stream of the [file agent proposal](README.md). It answers the open questions of [`../agent/05-sources-folder.md`](../agent/05-sources-folder.md) and keeps its decisions: link, don't copy; watch, then reconcile; re-read only on change; read in place; no writes. Facts were checked against the repo on 3 Oct 2026. Estimates are marked as such.
|
||||
|
||||
## Today
|
||||
|
|
@ -68,7 +70,7 @@ This is one stream of the [file agent proposal](README.md). It answers the open
|
|||
|
||||
**Electron and packaging**
|
||||
- There is no `dialog.showOpenDialog`. A dropped folder is reported as unsupported ([documents](../../architecture/documents.md#upload)).
|
||||
- The main process already calls the API: `watchImageModel` polls `/llm/image/local/runtime` on the loopback port every 5 s ([`main/index.ts`](../../../surfsense_local/electron/src/main/index.ts)). The sentence in [documents](../../architecture/documents.md#the-original-file) saying it never calls the backend is stale.
|
||||
- The main process already calls the API: `watchImageModel` polls `/llm/image/local/runtime` on the loopback port every 5 s ([`sidecars/watch-image-model.ts`](../../../surfsense_local/electron/src/main/sidecars/watch-image-model.ts)). The sentence in [documents](../../architecture/documents.md#the-original-file) saying it never calls the backend is stale.
|
||||
- Electron 44 has no `File.path`; `webUtils.getPathForFile` replaced it in Electron 32 ([breaking changes](https://www.electronjs.org/docs/latest/breaking-changes#removed-filepath)).
|
||||
- [`electron-builder.yml`](../../../surfsense_local/electron/electron-builder.yml) gives macOS an entitlements file and no `extendInfo`, so the app declares no `NS*UsageDescription` strings.
|
||||
- watchfiles 1.2.0 is locked only because `uvicorn[standard]` pulls it in ([`uv.lock`](../../../surfsense_local/backend/uv.lock), [`pyproject.toml`](../../../surfsense_local/backend/pyproject.toml)). [`api.spec`](../../../surfsense_local/backend/bundling/api.spec) collects uvicorn's submodules and nothing of watchfiles by name.
|
||||
|
|
@ -522,7 +524,7 @@ Nothing else branches.
|
|||
**Per-thread working folders.** `StorageSettings` gains `thread_working_dir(workspace_id, thread_id)` = `data/workspaces/<ws>/agent/threads/<thread>/`, holding `sources/` and `outputs/`.
|
||||
- `open_agent_session` creates the session there. `agent_turn` syncs, registers and streams there.
|
||||
- Thread delete nulls `output_path` on the thread's artifacts in its transaction, and removes the folder after the commit, beside `store.remove_thread`. `forget_sessions` gets the thread's folder.
|
||||
- Installers carry the agent since 7 Oct 2026, and any model may open an Agentic chat; M5 ships phases 3a and 3b.
|
||||
- Installers carry the agent since 7 Oct 2026, first released in 2.2.0, and any model may open an Agentic chat; phases 3a and 3b are built (below).
|
||||
- opencode's edit rule today, `*/agent/outputs/*`, matches no per-thread path. It becomes, last match winning:
|
||||
|
||||
```python
|
||||
|
|
@ -650,7 +652,7 @@ Milestones are the README's; durations there are rough.
|
|||
- Phase 0 ships first, in M0, as a 2.1.x point release that starts now. It does not wait for 02's clean-room work. It is a correctness fix with one plain column, and everything after it builds on it.
|
||||
- Phases 1 and 2 are M2. They need only M0 and run in parallel with M1 and M3.
|
||||
- Phase 5 is in M3, which also runs in parallel with M2; it lands once phase 1 is in.
|
||||
- Phases 3a and 3b are M5, after M2, M3 and M4.
|
||||
- Phases 3a and 3b were built ahead of M5, in 2c, and shipped in 2.2.0.
|
||||
- Phases 4a and 4b are M8, beside 04's RT3a, RT3b (Office support) and RT6, in parallel with M5 to M7.
|
||||
- The workspace and folder bundle (Export, import and sharing) is M6.
|
||||
|
||||
|
|
@ -781,6 +783,6 @@ These are integration tests under `tests/integration/` unless noted. Each lands
|
|||
4. Should instruction files inside a linked root be ignored by default, or offered later as an opt-in "folder rules" feature?
|
||||
5. Docling throughput on the reference laptops would set the first-scan threshold and the estimate shown when linking.
|
||||
6. Do Docling and opencode's Bun-based `read`, `grep` and `glob` handle `\\?\` paths and paths over 260 characters on Windows? The 259-character mirror cap assumes they do not.
|
||||
7. Does macOS attribute the API sidecar's file access to SurfSense's bundle, so its usage strings show? The packaged-app test in phase 4a answers it, and it needs a notarized build, which failed on 3 Oct 2026 for an expired Apple agreement. If not, the scan would have to start from a process macOS attributes correctly.
|
||||
7. Does macOS attribute the API sidecar's file access to SurfSense's bundle, so its usage strings show? The packaged-app test in phase 4a answers it, on a notarized build, as v2.1.0's and v2.2.0's are. If not, the scan would have to start from a process macOS attributes correctly.
|
||||
8. What share of a real Documents folder under OneDrive Known Folder Move is online-only? Measured in phase 4a on at least one business laptop, it sets the 20% threshold for the root's online-only line and the link dialog.
|
||||
9. Are 30 days of Trash and the 10-minute grant window right? Both are estimates, and trashed bytes keep using disk until purged.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
status: proposed
|
||||
status: in-progress
|
||||
code:
|
||||
- document_skills/
|
||||
- surfsense_local/backend/modules/agent/opencode_config.py
|
||||
|
|
@ -37,6 +37,8 @@ code:
|
|||
|
||||
> SurfSense's document work runs on deterministic engines that the app ships and no model can rewrite: a model writes a plan as data, an engine applies it to a copy of the file, checks the result and reports every operation. Agent-capable models reach the engines through typed SurfSense tools and learn the jobs from shipped SKILL.md files; smaller models get fixed workflows that call the same engines, with a skill the harness picks and a plan held to a JSON schema. The engines are the skills project's own code (`docxkit`, `crlib`, plan schemas, playbooks and graders). They ship in SurfSense after a scoped clean-room fix replaces the parts derived from Anthropic's proprietary skills, behind a no-regression eval gate and a provenance record that CI keeps true.
|
||||
|
||||
**Status, 11 Oct 2026:** the engines this page designs are not built: no `document_skills/`, no clean-room fix, no engines queue, no job threads, workflows or playbooks. What shipped in 2.2.0 covers part of its ground another way. Revised copies run SurfSense's own engines, written from ECMA-376, in the Studio worker ([03](03-editable-artifacts.md), [studio](../../architecture/studio.md#revised-copies)). Decision 9's loading is built: the agent loads four shipped SKILL.md files that name SurfSense tools and come with no script files, read-only through `skills.paths` and allowed by name, for every model rather than per model, with its shell denied ([agent](../../architecture/agent.md)). Since [#2184](https://github.com/MODSetter/SurfSense/pull/2184) no Studio format runs `exec()`: PowerPoint and Excel run their scripts in the document script runner and `runner.py` is deleted, so phase B's baseline can only be taken on that script path.
|
||||
|
||||
Facts below were checked on 3 Oct 2026 in this repo (`dev_mod` at `0847e12f7`), in opencode `1.18.32` source under `references/opencode-dev` (two patches behind the pinned `1.18.34`; marked *likely* where that matters), and in the skills project at `references/skills_to_improve` (branch `contract-redline-m1`: code at `f0bb40d`, developer docs at `5780aec`). `references/` is git-ignored, so those paths are named, not linked. Sibling streams: [01 sources and folders](01-sources-and-folders.md), [03 editable artifacts](03-editable-artifacts.md), [04 runtime and packs](04-runtime-and-packs.md), [05 model ladder and evals](05-model-ladder-and-evals.md) and [06 product shape](06-product-shape.md). Milestone numbers (M0 to M10) are the [README](README.md)'s.
|
||||
|
||||
## Today
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
status: proposed
|
||||
status: in-progress
|
||||
code:
|
||||
- surfsense_local/backend/alembic/versions/
|
||||
- surfsense_local/backend/modules/artifacts/
|
||||
|
|
@ -29,7 +29,7 @@ code:
|
|||
|
||||
> An artifact becomes a list of versions that only grows. A regenerate, a refine prompt, an edit to a selected passage, an accepted tracked change, a restore, a workflow run and a file the agent wrote each make a new version, and every version is created by one function and run by a named job on the Studio or the engines queue. The version shown and indexed, the head, moves only when a run succeeds, so a failed or cancelled run never hides the last good output. Every edit names the version it was made against and is refused when the head has moved. A filled questionnaire or a redlined contract is an artifact derived from the user's file: its first version is that file with the edits applied as tracked changes, the user's file is never written, and the copy downloads without internal comments unless the user asks for them. Every edit returns a per-operation report built by the engine that applied it, not by the model, and an edit the user did not ask to be partial either applies whole or saves nothing.
|
||||
|
||||
**Status, 6 Oct 2026:** on `slice/revise`, a cut of the revised copies is built: `surfsense_revise_document` makes v1 of a revised copy from a ticked `.docx`, `.xlsx`, `.xlsm` or `.pptx` source and its next versions, with all-or-nothing reports, Word edits always tracked, internal comments, Accept all and Reject all as versions, and the "With changes" and "Clean" downloads under localized names. Versions stay artifacts with `version` in metadata, not `artifact_versions` rows; the engines run in the Studio worker, not an engines queue; quotes replace `inspect_document` ids. What is true now, and what was cut, is in [studio](../../architecture/studio.md#revised-copies) and its Known gaps.
|
||||
**Status, 11 Oct 2026:** a cut of the revised copies is built, merged into `dev` with [#2194](https://github.com/MODSetter/SurfSense/pull/2194) and shipped in 2.2.0: `surfsense_revise_document` makes v1 of a revised copy from a ticked `.docx`, `.xlsx`, `.xlsm` or `.pptx` source and its next versions, with all-or-nothing reports, Word edits always tracked, internal comments, Accept all and Reject all as versions, and the "With changes" and "Clean" downloads under localized names. Versions stay artifacts with `version` in metadata, not `artifact_versions` rows; the engines run in the Studio worker, not an engines queue; quotes replace `inspect_document` ids. What is true now, and what was cut, is in [studio](../../architecture/studio.md#revised-copies) and its Known gaps.
|
||||
|
||||
Stream 03 of the file-agent proposal. Facts were checked against `dev_mod` at `0847e12f7` on 2026-10-03; "verified" means read in the linked file, "estimate" means not measured. Milestones (M0 to M10) are the [README](README.md)'s.
|
||||
|
||||
|
|
@ -461,7 +461,7 @@ They live in `modules/artifacts/version_router.py`. Quiz and flashcard routes pa
|
|||
| **6. Artifacts as sources** | M3 | Self-exclusion through 01's resolver; `inputs.sources[].version`; the newer-source hint. | 1; 01's scope resolver and filing (its phases 0 and 5) | S. |
|
||||
| **7. Keeping and forking** | M10 | Pin and configurable retention; Save as copy (`artifacts.derived_from_version`, kind `copy`); `markdown` and `csv` viewers with `allowedImagePrefixes: []`; PDF export through `export_job` with Office support; a filter on head title and body in Studio's list; cross-artifact dedup if measured disk use asks for it. | 1; 04's RT3b for PDF | M. |
|
||||
|
||||
Phases 2, 3 and 6 can run in parallel after 1. Phase 3 ships in M5 with 06's Answer / Work on files switch, and M5 turns the agent on in installers only once it has shipped, so files the agent writes are never invisible to the user. Phase 4 carries the B2B value; it waits on 02's clean-room engine work in M1 and ships with `docx.external` in M6.
|
||||
Phases 2, 3 and 6 can run in parallel after 1. Phase 3 ships in M5 with 06's Answer / Work on files switch, and M5 turns the agent on in installers only once it has shipped, so files the agent writes are never invisible to the user. The agent reached installers in 2.2.0 (9 Oct 2026) without phase 3, so a file it writes with its own tools stays in `outputs/`, outside Studio ([agent](../../architecture/agent.md#known-gaps)). Phase 4 carries the B2B value; it waits on 02's clean-room engine work in M1 and ships with `docx.external` in M6.
|
||||
|
||||
## Tests
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
status: proposed
|
||||
status: in-progress
|
||||
code:
|
||||
- surfsense_local/backend/pyproject.toml
|
||||
- surfsense_local/backend/bundling/worker.spec
|
||||
|
|
@ -29,7 +29,9 @@ code:
|
|||
|
||||
# Runtime and packs
|
||||
|
||||
> The document engines run inside the frozen Studio worker, which already carries almost every library they need, so the file agent adds about 75 MB to a Windows installer, most of it opencode, which installers carry from the release that turns the agent on (M5). The installer gains pypdf, three metric-compatible font families, a licence gate that also reads the native binaries inside the frozen tree, and a third-party notices file, and loses the 0.8 GB of developer caches that leak into local builds. LibreOffice is an optional runtime pack, shown in the interface as "Office support": a pinned upstream build, unaltered except for deleted files, that SurfSense re-hosts on its own releases, pins by sha256 in the app, and downloads only when the user asks and allows GitHub. It runs only outside the engine child and outside any database transaction, under a profile with updates, macros and links off, on a copy whose external links have been removed. Recalculated values never come from an alpha engine. A new ADR settles that executable packs are allowed after consent. No general Python ships for the engines or the agent's shell; the one the plugin system needs ships with plugins. No JavaScript runtime is needed, and Electron's RunAsNode fuse is switched off.
|
||||
> The document engines run inside the frozen Studio worker, which already carries almost every library they need, so the file agent adds about 75 MB to a Windows installer, most of it opencode, which installers carry since 2.2.0 (9 Oct 2026), ahead of M5. The installer gains pypdf, three metric-compatible font families, a licence gate that also reads the native binaries inside the frozen tree, and a third-party notices file, and loses the 0.8 GB of developer caches that leak into local builds. LibreOffice is an optional runtime pack, shown in the interface as "Office support": a pinned upstream build, unaltered except for deleted files, that SurfSense re-hosts on its own releases, pins by sha256 in the app, and downloads only when the user asks and allows GitHub. It runs only outside the engine child and outside any database transaction, under a profile with updates, macros and links off, on a copy whose external links have been removed. Recalculated values never come from an alpha engine. A new ADR settles that executable packs are allowed after consent. No general Python ships for the engines or the agent's shell; the one the plugin system needs ships with plugins. No JavaScript runtime is needed, and Electron's RunAsNode fuse is switched off.
|
||||
|
||||
**Status, 11 Oct 2026:** every installer carries opencode and ripgrep since 2.2.0 ([sizes](#installer-size-and-its-two-ceilings)). A cut of Office support (RT3b) shipped in the same release: the app downloads The Document Foundation's own build, untrimmed, from TDF's archive, pinned by sha256, after the user allows that host, or uses a LibreOffice the user confirms ([office pack](../../architecture/office-pack.md)). That departs from decisions 8 to 10: nothing is trimmed or re-hosted on SurfSense's releases, and the bytes come from whichever mirror TDF's archive redirects to. pypdf ships for the agent's PDF tools. RT0, RT2, RT3a's shared pack mechanism and ADR 0041, RT4, RT5 and RT6 are not built.
|
||||
|
||||
Facts below were checked on 3 Oct 2026 in this repo, in the local Windows build in `surfsense_local/electron/release/` (built 25 Sep 2026), and in upstream registries. Numbers are marked **measured** (with how) or **estimated**. Sibling streams: [01 sources and folders](01-sources-and-folders.md), [02 skills and engines](02-skills-and-engines.md), [03 editable artifacts](03-editable-artifacts.md), [05 model ladder and evals](05-model-ladder-and-evals.md), and [06 product shape](06-product-shape.md), which owns the interface names. Phases are numbered RT0 to RT6 so they do not collide with other streams' phase names, such as 06's R1. Milestones M0 to M10 are the [README](README.md)'s; the [Phases](#phases) table says which one carries each phase.
|
||||
|
||||
|
|
@ -38,10 +40,10 @@ Facts below were checked on 3 Oct 2026 in this repo, in the local Windows build
|
|||
### Installer size and its two ceilings
|
||||
|
||||
- `makensis` is 32-bit and cannot map a payload over 2 GB ([packaging](../../architecture/packaging.md#the-nsis-size-limit)). GitHub refuses any release asset of 2 GiB or more: "Each file included in a release must be under 2 GiB" ([GitHub docs](https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases)). The second ceiling applies to every platform and to anything SurfSense hosts on Releases.
|
||||
- v2.0.3's assets, measured by research report I7 with `gh release view`: `SurfSense-Setup.exe` 1,129,150,720 B, `SurfSense-arm64.dmg` 1,261,147,542 B, `SurfSense-arm64.zip` 1,245,087,801 B, `SurfSense.AppImage` 1,360,618,521 B, `SurfSense.deb` 1,246,171,516 B. The 2.1.0 dry run on 3 Oct uploaded a 1,130,260,665 B Windows artifact. Headroom is about 0.87 GB on Windows and 0.79 GB on the AppImage, the tightest. [packaging](../../architecture/packaging.md#the-nsis-size-limit) still quotes 2.0.2's 1.30 GB.
|
||||
- v2.0.3's assets, measured by research report I7 with `gh release view`: `SurfSense-Setup.exe` 1,129,150,720 B, `SurfSense-arm64.dmg` 1,261,147,542 B, `SurfSense-arm64.zip` 1,245,087,801 B, `SurfSense.AppImage` 1,360,618,521 B, `SurfSense.deb` 1,246,171,516 B. The 2.1.0 dry run on 3 Oct uploaded a 1,130,260,665 B Windows artifact. Headroom is about 0.87 GB on Windows and 0.79 GB on the AppImage, the tightest. *Since 9 Oct 2026:* v2.2.0's assets, with opencode and ripgrep inside, are `SurfSense-Setup.exe` 1,209,002,840 B, `SurfSense-arm64.dmg` 1,346,413,612 B, `SurfSense-arm64.zip` 1,327,395,136 B, `SurfSense.AppImage` 1,453,533,766 B and `SurfSense.deb` 1,335,815,512 B, 79 to 92 MB more than v2.1.0's with its other changes (`gh release view`), leaving about 0.79 GB on Windows and 0.69 GB on the AppImage; [packaging](../../architecture/packaging.md#the-nsis-size-limit) quotes its Windows installer.
|
||||
- The local `SurfSense-Setup.exe` of 25 Sep is 1,935,572,620 B (**measured**, `ls -l`), about 0.8 GB more than CI's. `electron-builder.yml` copies `from: ../backend/models` whole ([`electron-builder.yml`](../../../surfsense_local/electron/electron-builder.yml)). Locally that folder holds the three real packs (docling 398 MB, audio 181 MB, bge-small 65 MB) plus `hub/` 506 MB (a Docling Hugging Face cache), `kokoro/` 338 MB (the pre-audio.cpp voice), `xet/` and a dev `models.ini` (**measured**, `du -sm`). No step prunes them.
|
||||
- Differential updates are off, so every app update downloads the whole installer again ([updates](../../architecture/updates.md#consent)).
|
||||
- No installer carries opencode or ripgrep yet. `ENABLED_BY_DEFAULT` is `false` in [`enabled.mjs`](../../../surfsense_local/electron/scripts/opencode/enabled.mjs), whose comment ties the flip to "the agent test", and the "Stage opencode" step in [`release-local.yml`](../../../.github/workflows/release-local.yml) stages nothing while it is off. *Since 7 Oct 2026:* the maintainer turned it on ahead of M5, so every installer built since carries both, and release CI runs each installer's copy.
|
||||
- No installer carries opencode or ripgrep yet. `ENABLED_BY_DEFAULT` is `false` in [`enabled.mjs`](../../../surfsense_local/electron/scripts/opencode/enabled.mjs), whose comment ties the flip to "the agent test", and the "Stage opencode" step in [`release-local.yml`](../../../.github/workflows/release-local.yml) stages nothing while it is off. *Since 7 Oct 2026:* the maintainer turned it on ahead of M5, so every installer built since carries both, and release CI runs each installer's copy. v2.2.0, on 9 Oct 2026, is the first release that carries them.
|
||||
|
||||
### What the frozen worker already carries
|
||||
|
||||
|
|
@ -73,7 +75,7 @@ Facts below were checked on 3 Oct 2026 in this repo, in the local Windows build
|
|||
|
||||
- On a `v*` tag, electron-builder signs every `.exe` under `resources/` and no `.dll` or `.pyd` (I7, from the 2.1.0 dry-run log). The local worker holds 323 `.dll` and `.pyd` files, of which 89 carry a valid upstream signature, such as the PSF's on `python312.dll` (**measured**, `Get-AuthenticodeSignature`).
|
||||
- The macOS app uses the hardened runtime. `entitlementsInherit: build/entitlements.mac.plist` gives every nested binary `allow-jit`, `allow-unsigned-executable-memory`, `disable-library-validation` and `allow-dyld-environment-variables` ([`entitlements.mac.plist`](../../../surfsense_local/electron/build/entitlements.mac.plist), whose comment says the frozen sidecars and llama-server need the last two).
|
||||
- The 2.1.0 macOS dry run failed at notarization with Apple's 403 "A required agreement is missing or has expired" (I7, run 37102724459). The early credential check runs only on tag pushes ([`release-local.yml`](../../../.github/workflows/release-local.yml) "Verify Apple notarization credentials").
|
||||
- The 2.1.0 macOS dry run failed at notarization with Apple's 403 "A required agreement is missing or has expired" (I7, run 37102724459). The early credential check runs only on tag pushes ([`release-local.yml`](../../../.github/workflows/release-local.yml) "Verify Apple notarization credentials"). *Since 4 Oct 2026:* the account holder accepted the agreement, and that dry run's re-run on 4 Oct and the v2.1.0 and v2.2.0 tag builds logged "notarization successful", v2.2.0's with opencode inside.
|
||||
|
||||
### Licences
|
||||
|
||||
|
|
@ -112,7 +114,7 @@ The LibreOffice 25.2.7.2 installed on the development machine was copied into a
|
|||
7. **The running app may download pinned third-party runtime binaries, unaltered except for deleted files, after per-host consent, and a new ADR says so.** It reconciles [`cuda-backend.md`](../cuda-backend.md) with the agent README and the plugin design, and names executable downloads as a new class against the airgapped framing of ADR 0004, 0014 and 0017. The installer still downloads nothing, and every feature it ships works without a pack.
|
||||
8. **SurfSense CI builds packs and hosts them as releases in `SurfSense-Inc/surfsense-plugin-releases` under a `runtime-` tag prefix, with URLs, sizes and sha256s compiled into the app.** That repository will have the publisher App, its single-repository scope and the protected environment, so packs add no credential of their own. None of them exists today, so creating them is an RT3a task with named owners, following the plugin design's setup: an owner of SurfSense-Inc creates the repository and the App, and MODSetter creates the `plugin-publishing` environment and stores the App's ID and key. A `runtime-` release is never marked Latest, because the plugin catalogue refresh reads the Latest release. Re-hosting removes LibreOffice's mirror redirects; staying off `MODSetter/SurfSense` keeps pack releases out of the feed the updater reads; compiled-in pins need no remote catalogue and add no trust root.
|
||||
9. **A pack download follows exactly one redirect, `github.com` to `release-assets.githubusercontent.com`, and refuses any other before writing a byte.** Consent is per host ([ADR 0027](../../adr/0027-egress-consent-per-host.md)); a redirect elsewhere is egress nobody allowed.
|
||||
10. **The Office pack keeps LibreOffice's signed binaries untouched: Windows and Linux are trimmed by deletion only, and macOS ships TDF's `.app` whole.** Deletion keeps TDF's per-file Authenticode signatures (162 of 163 valid, measured). Any change inside a macOS bundle breaks its seal, and SurfSense cannot notarize a re-signed copy today.
|
||||
10. **The Office pack keeps LibreOffice's signed binaries untouched: Windows and Linux are trimmed by deletion only, and macOS ships TDF's `.app` whole.** Deletion keeps TDF's per-file Authenticode signatures (162 of 163 valid, measured). Any change inside a macOS bundle breaks its seal, and SurfSense would then re-sign and notarize every trimmed copy itself.
|
||||
11. **The pack pins the newest point release of the oldest TDF branch with at least six months of fixes left at the app's release, at `.1` or later.** Today that is 26.8.1: 26.2 ends on 30 Nov 2026. A pin must never start on an ended branch, and each app release re-applies the rule, so a LibreOffice security fix still needs an app release.
|
||||
12. **A LibreOffice the user already has is used only after they confirm it once in Settings › Downloads, and only if it is a supported TDF branch, carries no shared extensions and sits at a fixed install path.** Customer documents are untrusted input, an ended branch gets no security fixes, and a shared extension loads into every profile. It always runs under SurfSense's own profile, with updates off three ways and `LIBO_UPDATER_*` stripped (lessons 9 and 10).
|
||||
13. **One LibreOffice process runs at a time, app-wide, with a seeded profile, the caller's deadline, a bounded wait for the lock, macros and link updates off, proxies pointed at nowhere, and a copy whose external links are removed.** Three small files peaked at 400 MB (measured), Lambda guidance is at least 3,008 MB ([shelfio](https://github.com/shelfio/aws-lambda-libreoffice)), and llama-server is resident beside it. A shared profile collides on its lock. A DOCX or XLSX can pull another local file into a render through a `file:` link, a field or DDE, and a profile key is not a test.
|
||||
|
|
@ -468,9 +470,9 @@ The gate runs in `code-quality.yml` on pull requests that change a lockfile, a `
|
|||
|
||||
- **Windows, installer.** Every `.exe` is signed today, PBS's included from RT4. Adding `win.signExts: [".dll", ".pyd"]` waits on open question 4 (234 unsigned worker libraries, measured, against Smart App Control). Re-signing replaces an upstream signature, which 89 files carry, so a list would name only unsigned files.
|
||||
- **Windows, Office pack.** TDF's signatures survive the trim (measured), and SurfSense signs nothing in it. A file the backend's own HTTP client writes carries no Mark of the Web, so SmartScreen does not prompt (estimate, checked in RT3b on a clean VM).
|
||||
- **macOS, installer.** PBS's interpreter is signed by the `afterPack` hook with its own entitlements (section 4); its `.so` files are signed with the bundle. No notarized build has yet carried opencode or PBS. A notarized macOS build that carries opencode gates the agent's macOS release in M5: if notarization is still blocked then, the agent ships on Windows and Linux first, and the docs and the downloads page say so. A notarized build carrying PBS gates RT4 on macOS the same way.
|
||||
- **macOS, installer.** PBS's interpreter is signed by the `afterPack` hook with its own entitlements (section 4); its `.so` files are signed with the bundle. v2.2.0 is notarized and carries opencode under its vendor's signature (`mac.signIgnore`), so the agent ships on macOS as on the other two. No build has carried PBS yet, and a notarized build carrying it gates RT4 on macOS.
|
||||
- **macOS, Office pack.** TDF's `.app` is copied whole, so its signature and stapled ticket stay valid wherever it is extracted (verified in RT3b with `spctl`). A trimmed macOS pack needs SurfSense to re-sign and notarize it.
|
||||
- **The notarization blocker.** Every macOS release waits for the Apple Developer agreement to be accepted, which needs the account holder and no code. The README names its owner and a date to chase. RT0 runs the existing `notarytool history` step on manual dry runs too, and weekly on a schedule, so an expired agreement fails before a release rather than after a full build.
|
||||
- **The Apple Developer agreement.** Notarization stops whenever it lapses, as it did on 3 Oct 2026 until the account holder accepted it, which needs no code. RT0 runs the existing `notarytool history` step on manual dry runs too, and weekly on a schedule, so a lapsed agreement fails before a release rather than after a full build.
|
||||
|
||||
## Options considered and rejected
|
||||
|
||||
|
|
@ -485,7 +487,7 @@ The gate runs in `code-quality.yml` on pull requests that change a lockfile, a `
|
|||
| Using a found LibreOffice by default | It may be an ended branch or carry shared extensions, and running it is a choice about the user's own software |
|
||||
| A separate `github.com` consent for packs | Two rows for one host, and a per-host grant made for packs would silently allow every later errand on GitHub |
|
||||
| IronCalc values as a delivered workbook's cached values | An alpha engine's wrong value would be silent in every preview |
|
||||
| Trimming and re-signing the macOS `.app` now | Breaks TDF's seal, and SurfSense cannot notarize while the agreement is expired |
|
||||
| Trimming and re-signing the macOS `.app` now | Breaks TDF's seal, so SurfSense would re-sign and notarize every trimmed copy itself |
|
||||
| unoserver for throughput | Runs under LibreOffice's bundled Python, which the trim removes, and its README calls Windows "untested" (E5). One conversion at a time does not need it |
|
||||
| ONLYOFFICE x2t, PyMuPDF, HyperFormula, pycel, `formulas` | AGPL, AGPL, GPL-3, GPL-3, and EUPL plus scipy (E5) |
|
||||
| Typst for PDF output | 21.4 MiB on Windows (E5) for a path reportlab and the pack cover; revisit if PDF/A or PDF/UA becomes a job requirement |
|
||||
|
|
@ -511,7 +513,7 @@ The gate runs in `code-quality.yml` on pull requests that change a lockfile, a `
|
|||
| RT4 | Plugin interpreter | M10 | `scripts/python/`, `plugin_python()`, `entitlements.python.plist` and the signing hook, packaged smoke | the plugin runtime stream shipping | M: one more staged runtime, signed and notarized |
|
||||
| RT5 | CJK fonts pack | M10 | `fonts-cjk` on RT3a's mechanism; offer on CJK text | RT3a | S |
|
||||
|
||||
RT0 and RT2 are independent of every other stream. On this stream's side, turning the agent on in M5 needs only RT0's size gate, which then counts opencode and ripgrep, and the macOS notarization gate in section 10; the engine tools need RT1 in M6. 02 phase 5a's xlsx engine (M7) ships before the pack (M8): until then, and on any machine without the pack, workbooks carry cleared values the report names, and slides get no thumbnails. With 01 phases 4a and 4b (M8), this stream also takes 01's `mac.extendInfo` folder usage strings into `electron-builder.yml` and watchfiles into `api.spec` (section 2).
|
||||
RT0 and RT2 are independent of every other stream. On this stream's side, the agent reached installers in 2.2.0, ahead of M5 and before RT0's size gate exists, and that release met section 10's macOS notarization gate; the engine tools need RT1 in M6. 02 phase 5a's xlsx engine (M7) ships before the pack (M8): until then, and on any machine without the pack, workbooks carry cleared values the report names, and slides get no thumbnails. With 01 phases 4a and 4b (M8), this stream also takes 01's `mac.extendInfo` folder usage strings into `electron-builder.yml` and watchfiles into `api.spec` (section 2).
|
||||
|
||||
## Tests
|
||||
|
||||
|
|
@ -571,7 +573,7 @@ Unit and integration tests:
|
|||
- **[03 editable artifacts](03-editable-artifacts.md)**: the post-commit thumbnail job and where its PNGs live; the PDF export route that enqueues `export_pdf()`; chunks and vectors for engine versions computed outside the engines worker; the xlsx diff's "values not recalculated" note matching decision 17's report lines; the DOCX viewer's webfont mapping.
|
||||
- **[05 model ladder and evals](05-model-ladder-and-evals.md)**: the Anthropic SDK in both binaries with P1 in M4 (about 2 MB of wheels, within budget); the `agent_smoke` and `free_agent_tasks` rows on the native Claude route (`ENABLED_BY_DEFAULT` was turned on without them on 7 Oct 2026); the llama.cpp bump (P5a) as a pin change under the size gate; the opencode pin policy (decision 20); the matrix running on the packaged worker and staged opencode.
|
||||
- **[01 sources and folders](01-sources-and-folders.md)**: legacy formats in linked folders, converted at ingest from RT6, and the root header's count of unreadable legacy files until then; `@tanstack/react-virtual` (MIT) passes the gate; watchfiles in `api.spec` and the `mac.extendInfo` strings with phases 4a and 4b.
|
||||
- **[README](README.md)**: the whole-stack memory budget, measured at the M5 and M6 exits, that counts the engines worker and, from M8, LibreOffice; owners and chase dates for the Apple agreement and TDF's answer.
|
||||
- **[README](README.md)**: the whole-stack memory budget, measured at the M5 and M6 exits, that counts the engines worker and, from M8, LibreOffice; the owner and chase date for TDF's answer.
|
||||
- **[06 product shape](06-product-shape.md)**: the names "Office support" and "Needs Office support", Settings › Downloads as their place, and "pack" in the interface kept for job content.
|
||||
- **Plugin streams**: the plugin runtime shipping (RT4). The plugin releases repository, its publisher App and the `plugin-publishing` environment are created by whichever of RT3a and the plugin installer starts first (section 6).
|
||||
|
||||
|
|
@ -581,7 +583,7 @@ Unit and integration tests:
|
|||
2. Is TDF's macOS aarch64 build notarized with a stapled ticket, so that an untouched `.app` extracted into the data folder passes `spctl`? RT3b's smoke answers it.
|
||||
3. Does LibreOffice on Windows load fonts from the pack's `share/fonts/truetype/`, or must they sit in the profile or the system? This decides whether the Windows pack needs the fonts copied elsewhere.
|
||||
4. Does Windows 11 Smart App Control block the installer's 234 unsigned worker libraries, or the pack's one unsigned `fbintl.dll`? Test on a SAC-enabled VM before choosing `win.signExts`.
|
||||
5. Who accepts the Apple Developer agreement, and when? Nothing in this proposal ships on macOS before that, and the agent's macOS release waits on it (section 10). The README names the owner and a chase date.
|
||||
5. ~~Who accepts the Apple Developer agreement, and when?~~ Answered: the account holder accepted it by 4 Oct 2026, when the 2.1.0 dry run's re-run was notarized; v2.1.0 and v2.2.0 are notarized, and v2.2.0 carries the agent on macOS.
|
||||
6. Does the updater still install N to N+1 on all three platforms once the RunAsNode fuse is off?
|
||||
7. How much of a real security questionnaire's formulas does IronCalc 0.8.3 evaluate, and how often does it disagree with LibreOffice? 02 measures it on the questionnaire fixtures before adopting it.
|
||||
8. Do `ctypes` callbacks in PBS 3.12 on macOS arm64 need `allow-jit` or `allow-unsigned-executable-memory` under the hardened runtime? The RT4 smoke answers it; the entitlements file grows only if it fails.
|
||||
|
|
|
|||
|
|
@ -34,11 +34,11 @@ code:
|
|||
|
||||
# Model ladder and evals
|
||||
|
||||
**Status, 4 Oct 2026:** a first cut of P4a and P4b's rows is built on `slice/capability` ([model capabilities](../../architecture/model-capabilities.md)). `capability_of(model, connection)` gives each chat model one of four levels from a list shipped with the app, written by `scripts/write_capability_list.py` from the first ladder's screening results (11 models, one run per cell, marked provisional); the engine choice and Studio's Word and PDF path follow it, `TESTED_MODELS` is gone, and, since 7 Oct 2026, every model may run the agent in a chat opened in Agentic mode, the level setting only the default. It differs from section 4: the list is `capability/measured/capabilities.json`, keyed by a canonical model id and where the model was served rather than path, build and window; `capability_of` reads no database and has no `apply_org_policy()`, confirmations, format availability or rungs; and the tier still comes from `classify()`. P0 to P3 and P5 to P7 are not built.
|
||||
**Status, 11 Oct 2026:** a first cut of P4a and P4b's rows is built, reached `dev` with [#2177](https://github.com/MODSetter/SurfSense/pull/2177) and shipped in 2.2.0 ([model capabilities](../../architecture/model-capabilities.md)). `capability_of(model, connection)` gives each chat model one of four levels from a list shipped with the app and marked provisional: rows that `scripts/write_capability_list.py` writes from the first ladder's screening results (one run per cell), rows that `scripts/catalog_rows.py` wrote from a two-case screening sweep over OpenRouter on 7 and 8 Oct 2026, and flagships assumed without runs ([model capabilities](../../architecture/model-capabilities.md#the-list), [07-dev-setup](07-dev-setup.md#live-tests)); the engine choice and Studio's Word and PDF path follow it, `TESTED_MODELS` is gone, and, since 7 Oct 2026, every model may run the agent in a chat opened in Agentic mode, the level setting only the default. Like the ladder's, the sweep's rows are screening, not committed rows. It differs from section 4: the list is `capability/measured/capabilities.json`, keyed by a canonical model id and where the model was served rather than path, build and window; `capability_of` reads no database and has no `apply_org_policy()`, confirmations, format availability or rungs; and the tier still comes from `classify()`. P0 to P3 and P5 to P7 are not built.
|
||||
|
||||
> SurfSense learns what each model can do by measuring it, never by its size or its prompt tier. Claude reaches the app through Anthropic's own Messages API, with prompt caching, enforced schemas and a spend ledger, so the first column of the matrix measures Claude rather than a compatibility layer. A job × model matrix, run at the app's own settings with the skills project's graders, measures the jobs and the free-form file work users are offered, walks down from Opus to a 4B local model, and holds one OpenAI and one Gemini column beside Claude. Each result lands in a reviewed capability manifest shipped with the app. One function, `capability_of()`, reads that manifest and returns the capability that the product shape in [06](06-product-shape.md) shows: which engine new threads get, how each job runs, which Studio formats and edit rungs are offered, whether the agent may ask for the shell, and the window it needs. An unmeasured model keeps everything it has today, labelled "not measured"; only a measured failure takes something away.
|
||||
|
||||
**Status, 4 Oct 2026:** proposed. A first screening row exists: the [create-and-edit slice](07-create-and-edit-mvp.md)'s eight live cases, run once per cell through Anthropic's compatibility layer, pass on Opus 5.5 and Sonnet 5.5, and five of eight pass on Haiku 4.5 ([08](08-model-ladder-results.md)). Eight open models were screened through OpenRouter on the hosts it chose. Kimi K3 and Qwen3.8-27B pass all eight cases. GLM-5.3 and DeepSeek V4 Pro, both text-only, pass all but the figure swap. Qwen3.6-35B-A3B, Gemma 4 31B, Ministral 14B and Qwen3.5-9B fall short, mostly by not checking their pages ([08](08-model-ladder-results.md#open-models-through-openrouter)). The 8–14B tier on llama.cpp is not run yet. The live tests choose their model and provider (Anthropic or OpenRouter) from the environment ([07-dev-setup](07-dev-setup.md#live-tests)), which is how the open-model columns were screened. None of these are committed rows: n=1, the dev app, no native provider.
|
||||
**Status, 4 Oct 2026:** the matrix is not built. A first screening row exists: the [create-and-edit slice](07-create-and-edit-mvp.md)'s eight live cases, run once per cell through Anthropic's compatibility layer, pass on Opus 5.5 and Sonnet 5.5, and five of eight pass on Haiku 4.5 ([08](08-model-ladder-results.md)). Eight open models were screened through OpenRouter on the hosts it chose. Kimi K3 and Qwen3.8-27B pass all eight cases. GLM-5.3 and DeepSeek V4 Pro, both text-only, pass all but the figure swap. Qwen3.6-35B-A3B, Gemma 4 31B, Ministral 14B and Qwen3.5-9B fall short, mostly by not checking their pages ([08](08-model-ladder-results.md#open-models-through-openrouter)). The 8–14B tier on llama.cpp is not run yet. The live tests choose their model and provider (Anthropic or OpenRouter) from the environment ([07-dev-setup](07-dev-setup.md#live-tests)), which is how the open-model columns were screened. None of these are committed rows: n=1, the dev app, no native provider.
|
||||
|
||||
## Today
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
status: proposed
|
||||
status: in-progress
|
||||
code:
|
||||
- surfsense_local/backend/alembic/versions/<next>_thread_artifact.py
|
||||
- surfsense_local/backend/alembic/versions/<next>_workspace_settings_and_thread_jobs.py
|
||||
|
|
@ -34,6 +34,8 @@ code:
|
|||
|
||||
> SurfSense has one kind of workspace. The user never picks a mode. They pick a model, and what the app offers inside every workspace follows from that model's measured capability profile ([05](05-model-ladder-and-evals.md)): every model answers questions and makes Studio outputs, models that pass the job tests run jobs step by step or as a job thread, and models that pass the agent tests can also work on files freely. Jobs (study pack, contract redline, security questionnaire, RFP response) are listed on every model, and each one says how it will run on the selected model, or what it needs and the cheapest way to get it. Folders can carry job roles, a workspace can be marked local only, and a job's output can be marked reviewed. None of that is a workspace type. Job content is open source and ships in the app; a license adds premium plugins, support and Enterprise controls, and never changes what a model may do.
|
||||
|
||||
**Status, 11 Oct 2026:** a cut of S3 shipped in 2.2.0 ([#2207](https://github.com/MODSetter/SurfSense/pull/2207)), and it departs from this page. The composer offers Basic (Q&A) and Agentic for each new chat, so the interface does say "mode". Every model may pick Agentic unless a technical gate keeps it out: a stated no on tool calls, a window under 32,768 tokens, or no opencode. A measured level sets only the default, a user's pick is remembered per model, and a chat keeps its mode. `ThreadCreate` takes `mode` and `remember`; a gated request is refused with `409` and its code, and one whose agent does not start with `503` ([model capabilities](../../architecture/model-capabilities.md#the-modes), [agent](../../architecture/agent.md#which-threads-get-it)). Verdict labels, jobs, workspace settings, handoff and review are not built.
|
||||
|
||||
Stream 06 of the file-agent proposal. It turns [05](05-model-ladder-and-evals.md)'s `Capability` into what the user sees and owns the job catalogue that [02](02-skills-and-engines.md) and [03](03-editable-artifacts.md) wait on. Facts were checked against `dev_mod` at `0847e12f7` on 2026-10-03. Milestone numbers (M0 to M10) are the [README](README.md#milestones-in-order)'s. Both design-panel judges chose P1 (48) over workspace types (P2, 38–41) and user-picked modes (P3, 40); the grafts taken are under Options considered.
|
||||
|
||||
## The answer to the maintainer's question
|
||||
|
|
@ -123,7 +125,7 @@ The 4B user's screen therefore never shows an agent control, and the frontier us
|
|||
### Threads: Answer and Work on files
|
||||
|
||||
- `frontend/src/features/chat/engine-switch.tsx` (new) is a two-state control in an empty thread's composer. It renders only when `"agent"` is in `capability.engines` or the agent verdict is `untested` with every live gate passing.
|
||||
- Release builds carry opencode since `ENABLED_BY_DEFAULT` turned true on 7 Oct 2026, ahead of 05's `agent_smoke` and `free_agent_tasks` rows and 03 phase 3. If macOS notarization is still blocked then, the agent ships on Windows and Linux first and the docs say so.
|
||||
- Release builds carry opencode since `ENABLED_BY_DEFAULT` turned true on 7 Oct 2026, ahead of 05's `agent_smoke` and `free_agent_tasks` rows and 03 phase 3. v2.2.0 (9 Oct 2026) is the first release that carries it, on all three platforms, notarized on macOS.
|
||||
- The default is `capability.default_engine`, which 05 sets to `agent` only on paths that cache prompts (`anthropic-native`, `llamacpp`) and to `chat` elsewhere. Whether a cached route should default to Work on files is the README's [maintainer question 7](README.md#open-questions-for-the-maintainer): new threads default to Answer until M4 measures first-answer latency and cost on Sonnet, and move to 05's rule only if Work on files is no slower for a one-line question. The last choice per workspace is remembered in `localStorage` (`surfsense:new-thread-engine:v1`, failures ignored), and only when the capability still allows it.
|
||||
- The first send carries `ThreadCreate.engine`. After that the control becomes the header label: a thread keeps its engine ([agent README](../agent/README.md), Locked decisions).
|
||||
- Choosing "Work on files" on an unconfirmed untested model shows an inline notice: "SurfSense has not measured working on files with {model}. Results may be wrong or incomplete, and on an API key each step costs tokens." Confirming calls `POST /llm/capability/confirmations` (05); Settings › Models can revoke it.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
> A user asks the agent in chat for a Word or PDF document, sees it appear in Studio, and refines it turn by turn: each turn makes a new version, and the user can switch between versions. The agent writes the document as a Python script, SurfSense runs it without asking and keeps it, and the next edit changes the script. Documents carry images from the user's sources and charts. Studio's own Word and PDF buttons stay as they are today. The first file is never the user's own source, and nothing promises to keep a source's styles. Giving Studio a script path for strong models and a Markdown path for small ones, each with a one-call Refine, comes [after the slice](#after-the-slice).
|
||||
|
||||
**Status, 4 Oct 2026:** S1, S2, S3, S7, S8 and S9 are built on `dev_mod`, and S6 is [07-dev-setup](07-dev-setup.md); the three-turn demo passes live against Claude Sonnet 5.5. On `slice/agent-formats`, not yet merged into `dev_mod`, the same tool also makes PowerPoint decks and Excel workbooks, a Word document or deck can start from a ticked source's own `.docx` or `.pptx` (`template_source_id`), decks get slide previews as Word gets pages, a workbook's result carries its summary instead, and `surfsense_source_pages` draws a PDF, Word or PowerPoint source's pages for the agent to match a look. What is true now is in [agent](../../architecture/agent.md), [studio](../../architecture/studio.md#script-documents) and [documents](../../architecture/documents.md#figures), with their Known gaps. After the slice, S4, S5 and the code path with its Refine are built on `slice/free-tier` ([studio](../../architecture/studio.md#word-and-pdf)), tested end to end with scripted models but not yet run against a live one. Two choices there differ from decision 7: which models are strong is a provisional rule, a model served from this computer (llama.cpp, or a connection on loopback such as Ollama or LM Studio) small and a remote one strong; and a strong model's first draft is shown its script's error and asked again, up to three calls, while Refine stays one call. Refine is Studio's only, as decision 8 says: the agent's documents are edited in its chat. Studio's own PowerPoint and Excel still run on `exec()`. In the slice itself, two choices differ from below: the tool server's timeout is 200 seconds, so a render's 150-second wait and a 30-second Word preview fit in one 190-second call deadline (decision 5), and a PDF's figures are cropped from its pages with pypdfium2 instead of Docling's picture images, which held every page's render in memory (decision 10). Left: merging into `dev`, measuring matplotlib against the installer's size gate, measuring the two paths on live models, and moving Studio's own PowerPoint and Excel off `exec()`.
|
||||
**Status, 11 Oct 2026:** S1, S2, S3, S7, S8 and S9 are built, and S6 is [07-dev-setup](07-dev-setup.md); the three-turn demo passes live against Claude Sonnet 5.5. The same tool also makes PowerPoint decks and Excel workbooks, a Word document or deck can start from a ticked source's own `.docx` or `.pptx` (`template_source_id`), decks get slide previews as Word gets pages, a workbook's result carries its summary instead, and `surfsense_source_pages` draws a PDF, Word or PowerPoint source's pages for the agent to match a look. What is true now is in [agent](../../architecture/agent.md), [studio](../../architecture/studio.md#script-documents) and [documents](../../architecture/documents.md#figures), with their Known gaps. After the slice, S4, S5 and the code path with its Refine are built ([studio](../../architecture/studio.md#word-and-pdf)), tested end to end with scripted models but not yet run against a live one. Two choices there differ from decision 7: which models are strong is a provisional rule, a model served from this computer (llama.cpp, or a connection on loopback such as Ollama or LM Studio) small and a remote one strong; and a strong model's first draft is shown its script's error and asked again, up to three calls, while Refine stays one call. Refine is Studio's only, as decision 8 says: the agent's documents are edited in its chat. Since [#2184](https://github.com/MODSetter/SurfSense/pull/2184), Studio's own PowerPoint and Excel scripts run in the script runner too, and `office/runner.py` is deleted. In the slice itself, two choices differ from below: the tool server's timeout is 200 seconds, so a render's 150-second wait and a 30-second Word preview fit in one 190-second call deadline (decision 5), and a PDF's figures are cropped from its pages with pypdfium2 instead of Docling's picture images, which held every page's render in memory (decision 10). All of it except #2184's change reached `dev` with [#2177](https://github.com/MODSetter/SurfSense/pull/2177) on 5 Oct 2026 and shipped in 2.2.0 on 9 Oct; #2184 merged on 11 Oct, after the release. Left: measuring matplotlib against the installer's size gate, and measuring the two paths on live models.
|
||||
|
||||
This slice is built first, on `dev_mod`, which is synced with `dev` every few hours and merged into `dev` when the slice is ready, ahead of the milestones in the [README](README.md). It follows [ADR 0039](../../adr/0039-document-scripts-run-without-approval.md). Facts were checked against `dev_mod` at `0847e12f7` on 3 Oct 2026.
|
||||
|
||||
|
|
@ -53,7 +53,7 @@ If a script fails three times in a row, the agent stops and tells the user what
|
|||
|
||||
*Reason:* code-written Word and PDF is proven today, and strong models write that code well. Writing Markdown is a single call that small models do well, and the builders return ADR 0010's rule to that path.
|
||||
8. **After the slice, Refine is one call on either path.** `POST /artifacts/{id}/refine` takes `{instruction}`, creates the next version as a pending artifact, and runs a Studio job. On the Markdown path, the job sends the current Markdown and the instruction, gets the whole revised Markdown back, and renders it. On the code path, it sends the current script and the instruction, gets the whole revised script back, and runs it through the runner. Nothing is patched or diffed. A spec longer than the selected model's window is refused with a reason. In this slice, the agent's documents are refined in chat.
|
||||
9. **The video runs from a development build.** Connect Claude with the user's Anthropic key as an OpenAI-compatible connection; Sonnet 5.5 passed the ladder, so its new chats start in Agentic. Installers built since 7 Oct 2026 carry the agent too.
|
||||
9. **The video runs from a development build.** Connect Claude with the user's Anthropic key as an OpenAI-compatible connection; Sonnet 5.5 passed the ladder, so its new chats start in Agentic. Installers built since 7 Oct 2026 carry the agent too. The video the root README shows, added on 9 Oct 2026, is a contract redline instead ([06-repo-readme](../../../plans/community-local/seo/06-repo-readme.md#the-readme-since-9-oct-2026)).
|
||||
10. **Ingestion keeps each source's figures.** Docling's layout model already finds the figures on a PDF page, drawn charts and figures in scans included, with their captions. Its Word and PowerPoint readers return the pictures those files embed.
|
||||
- **Turning it on:** `parsing.py` turns on `generate_picture_images` at an `images_scale` of 2. After converting, it saves each figure as a PNG with its caption, page and size, before exporting the Markdown.
|
||||
- **Where figures live:** beside the source's original, in `figures/<n>.png` with a `figures.json` index, inside the document's own folder. Deleting the source deletes them, and no table or migration is needed.
|
||||
|
|
|
|||
|
|
@ -112,7 +112,7 @@ Claude is priced from [`model_prices.py`](../../../surfsense_local/backend/tests
|
|||
|
||||
**What a run leaves.** Each run writes `references/live-runs/<UTC time>-<case>-<provider>-<model>/`: `transcript.md` (messages, tool calls and results, redacted), `frames.json` (the thread's frames), `model-requests.json` (each request's usage, whether it carried an image, where it went with the headers it carried, key redacted, and for OpenRouter which provider served it and what it billed), the generated `.docx` and `.pdf` of every version with its script, the sources, the preview PNGs, `cost.json` (with the model, provider and prices, and `reported_dollars`, OpenRouter's own bill when every request carried one) and `result.json` (with the model, provider, whether the app declared image input, and the `metrics` a case grades without failing on). Open these to see what the agent did and made.
|
||||
|
||||
**The screening sweep.** [`tests/live/sweep/`](../../../surfsense_local/backend/tests/live/sweep/) widens the capability list beyond the 8-case ladder. From an OpenRouter `/models` listing it keeps every model that takes `tools`, holds 32,768 tokens, has prices and is a fixed model (no `:free` or `:batch` tier, no `~` alias, no `openrouter/` router), one per `model_key`, its cheapest variant. Keys the list already measured, on the ladder or in an earlier sweep, keep their rows; an assumed flagship is chosen again, and screened once its price falls under $3/M. Models with input at $3/M and up are not run but listed as assumed to pass, and Opus and Fable models are never run. On the 7 Oct 2026 listing that is 242 models to run and 29 assumed. Each model runs smoke, then the PDF brief and the board pack: a smoke it fails stops it at 0 of 2, passing both cases is `agent` and anything less is `below`, and when exactly one case fails it is run once more and that run decides it. A provider fault (a 408, 429 or 5xx, an overload, a connection that dropped or timed out, no provider up for the model, a time-out with no output) or a harness fault (no `result.json`, a crash, an app error, the thread opening as a chat) is tried up to twice more and never counted; three in a row leave the case unresolved, which `--retry-unresolved` reopens. OpenRouter refusing the account (a 401 or 402: a bad key, no credits) is never counted either, and stops the sweep until the same command resumes it; `report` says why it stopped.
|
||||
**The screening sweep.** [`tests/live/sweep/`](../../../surfsense_local/backend/tests/live/sweep/) widens the capability list beyond the 8-case ladder. From an OpenRouter `/models` listing it keeps every model that takes `tools`, holds 32,768 tokens, has prices and is a fixed model (no `:free` or `:batch` tier, no `~` alias, no `openrouter/` router), one per `model_key`, its cheapest variant. Keys the list already measured, on the ladder or in an earlier sweep, keep their rows; an assumed flagship is chosen again, and screened once its price falls under $3/M. Models with input at $3/M and up are not run but listed as assumed to pass, and so is each id `--assume <id> ...` names, whatever its price. Opus and Fable models are never run, `--only` included. On the 7 Oct 2026 listing that is 242 models to run and 29 assumed; the sweep named 19 more with `--assume`, flagships whose output costs $10/M or more, so it ran 223 and assumed 48. The list keeps no separate record of those names: an `assumed` row is taken as unmeasured, so a later sweep screens every assumed model now under $3/M of input, the 19 and the four stopped at `--case-dollars` among them, unless `--assume` names them again; each `assumed` row of `capabilities.json` carries its `model_id`, and an id the listing lacks is refused. Each model runs smoke, then the PDF brief and the board pack: a smoke it fails stops it at 0 of 2, passing both cases is `agent` and anything less is `below`, and when exactly one case fails it is run once more and that run decides it. A provider fault (a 408, 429 or 5xx, an overload, a connection that dropped or timed out, no provider up for the model, a time-out with no output) or a harness fault (no `result.json`, a crash, an app error, the thread opening as a chat) is tried up to twice more and never counted; three in a row leave the case unresolved, which `--retry-unresolved` reopens. A retry waits a minute after the fault, and six harness faults in a row, across models, stop the sweep. A request the provider refuses as longer than the model's window counts as a failure. OpenRouter refusing the account (a 401 or 402: a bad key, no credits) is never counted either, and stops the sweep until the same command resumes it; `report` says why it stopped.
|
||||
|
||||
```bash
|
||||
cd surfsense_local/backend
|
||||
|
|
@ -121,10 +121,21 @@ OPENROUTER_API_KEY="$(powershell.exe -NoProfile -Command "[Environment]::GetEnvi
|
|||
uv run python -m tests.live.sweep report
|
||||
```
|
||||
|
||||
`--fetch` takes a fresh listing in place of `--listing`, and `--only <id> ...` runs exactly the ids named, measured or not. Each case is its own pytest process in a lane with its own data and temp dirs and its own block of ports ([`lane_ports.py`](../../../surfsense_local/backend/tests/live/lane_ports.py), `SURFSENSE_LIVE_PORT_BASE`), so two lanes never bind the same port; a model's two cases may run at once. A RAM guard starts a case only while free memory, less what running cases may still take, leaves 2 GB (`--min-free-gb`); a case peaked at 0.9 to 1.4 GB in the 7 Oct 2026 pilot, and the guard reserves 1.6 GB for each, so six lanes want about 12 GB free. `report` gives its ETA at the lanes that were actually busy. `--budget` is a hard cap on starting: a case starts only if what is spent, the cases in flight and its own expected cost fit under it, and each run's stop is `--case-dollars` ($3), which ends a runaway model, or what the sweep has left. `--case-minutes` (20) kills a run that goes on longer. One Ctrl-C starts nothing new and lets the cases in flight finish, since each runs in its own process group; a second kills them, and a restart runs them again. Everything lands in `references/live-runs/sweep/`: `attempts.jsonl`, one line per attempt, which a restarted sweep reads to skip what is done (a `run` on a folder another runner still holds exits, naming its pid, rather than pay for its cases again); each model's folder with its runs and `summary.json`; and `sweep-results.json` for the capability list's generator: under `models` only the models with a verdict, since the generator reads each one's counts as a measurement, under `unfinished` those unresolved or not yet done, then the assumed list and the `date` of the last counted run.
|
||||
`--fetch` takes a fresh listing in place of `--listing`, and `--only <id> ...` runs exactly the ids named, measured or not. Each case is its own pytest process in a lane with its own data and temp dirs and its own block of ports ([`lane_ports.py`](../../../surfsense_local/backend/tests/live/lane_ports.py), `SURFSENSE_LIVE_PORT_BASE`), so two lanes never bind the same port; a model's two cases may run at once. Before and after each case the runner stops whatever is left running from the lane's folder, or an opencode on its ports (`reap_lane` in [`runner.py`](../../../surfsense_local/backend/tests/live/sweep/runner.py)): a case that died left its opencode holding the lane's database, and every later case in that lane failed before it started. A RAM guard starts a case only while free memory, less what running cases may still take, leaves 2 GB (`--min-free-gb`); a case peaked at 0.9 to 1.4 GB in the 7 Oct 2026 pilot, and the guard reserves `--lane-gb` (1.6 GB) for each, so six lanes want about 12 GB free; after ten cases it reserves the 95th percentile of the peaks it saw, never less than 1.6 GB, or than `--lane-gb` when that is smaller. `report` gives its ETA at the lanes that were actually busy. `--budget` is a hard cap on starting: a case starts only if what is spent, the cases in flight and its own expected cost fit under it, and each run's stop is `--case-dollars` ($3) or what the sweep has left. A case stopped at `--case-dollars` ends its model as a flagship too dear to test, listed as assumed to pass (four were in the sweep of 7 and 8 Oct 2026); one stopped at the sweep's last dollars is not counted and stops the sweep. `--case-minutes` (20) kills a run that goes on longer, which counts as failed when the model wrote anything. One Ctrl-C starts nothing new and lets the cases in flight finish, since each runs in its own process group; a second kills them, and a restart runs them again. Everything lands in `--out`, by default `references/live-runs/sweep/` (`sweep/` under `SURFSENSE_LIVE_RUNS_DIR` when that is set): `listing.json`, the listing it chose from, so a resumed run needs neither `--listing` nor `--fetch`; `models.json`, the models it chose, which `--dry-run` writes before it stops; `runner.log`, each decision with the free memory then; `attempts.jsonl`, one line per attempt, which a restarted sweep reads to skip what is done (a `run` on a folder another runner still holds exits, naming its pid, rather than pay for its cases again); each model's folder with its runs and `summary.json`; and `sweep-results.json` for the capability list's generator: under `models` only the models with a verdict, since the generator reads each one's counts as a measurement, under `unfinished` those unresolved or not yet done, then the assumed list and the `date` of the last counted run.
|
||||
|
||||
**From the sweep to the list.** Once the sweep is done, add its rows to the list the app ships:
|
||||
|
||||
```bash
|
||||
cd surfsense_local/backend
|
||||
PYTHON_DOTENV_DISABLED=1 PYTHONPATH=. uv run python scripts/catalog_rows.py ../../references/live-runs/sweep/sweep-results.json
|
||||
```
|
||||
|
||||
It rewrites `capabilities.json` and prints each row it added, replaced or left out, and each model left out for want of a verdict ([model capabilities](../../architecture/model-capabilities.md#the-list)); review the diff, bring that doc's counts in line, and commit both. To measure again models the list already screened, such as after a fix in the tools, run them with `--only` into a new `--out`, since a folder's `attempts.jsonl` already holds their verdicts, and delete their rows from the list before adding the re-run's: the script keeps a listed row over an added one of its suite. The 16 OpenAI models an optional id of 0 had failed, and two Grok models, ran again this way, into their own folder ([agent](../../architecture/agent.md#surfsenses-tools)). Both runs are committed under [`capability_list/sweep/runs/`](../../../surfsense_local/backend/scripts/capability_list/sweep/runs/), and the list's rows come from [`openrouter-screen-v1.json`](../../../surfsense_local/backend/scripts/capability_list/sweep/openrouter-screen-v1.json) beside them: the screen with a model the re-run finished taking the re-run's row, one it left unfinished dropped, and the re-run's capped models added to `assumed`. With only the ladder's rows in `capabilities.json`, `catalog_rows.py` on that file writes the shipped list back unchanged ([#2221](https://github.com/MODSetter/SurfSense/issues/2221)).
|
||||
|
||||
## The demo video
|
||||
|
||||
The video in the root README is not this script. Added on 9 Oct 2026 ([#2212](https://github.com/MODSetter/SurfSense/pull/2212)), it shows the agent redlining a city contract as a revised copy, rehearsed on Qwen3.8 27B through OpenRouter; its source file, query and rehearsal are in [06-repo-readme](../../../plans/community-local/seo/06-repo-readme.md#the-readme-since-9-oct-2026). The three turns below are the live test's demo flow (`test_demo_flow.py`).
|
||||
|
||||
Record from the dev build above. One run of the three turns costs about $1.50 with Sonnet 5.5 and takes a little over three minutes, most of it the model.
|
||||
|
||||
**Before recording:** the three sources ready in the sources panel; Sonnet selected; Studio open in the right rail with an empty artifact list; a fresh agent thread. Keep the window wide enough that the chat and Studio show side by side. Do not reload the thread while recording: a `[n]` the agent wrote for a file it read without searching streams but is dropped from the saved reply ([agent](../../architecture/agent.md#known-gaps)).
|
||||
|
|
|
|||
|
|
@ -396,7 +396,7 @@ No run failed the preview check. Every render that drew pages had them matched b
|
|||
|
||||
## Still to run
|
||||
|
||||
The columns are those of 05's [eval matrix](05-model-ladder-and-evals.md#5-the-eval-matrix). Each runs the same eight cases with `SURFSENSE_LIVE_PROVIDER` and `SURFSENSE_LIVE_MODEL` ([07-dev-setup](07-dev-setup.md#live-tests)).
|
||||
The columns are those of 05's [eval matrix](05-model-ladder-and-evals.md#5-the-eval-matrix). Each runs the same eight cases with `SURFSENSE_LIVE_PROVIDER` and `SURFSENSE_LIVE_MODEL` ([07-dev-setup](07-dev-setup.md#live-tests)). A screening sweep over OpenRouter on 7 and 8 Oct 2026 ran 208 more models through a smoke check, the PDF brief and the board pack only, Qwen3.5-397B-A17B, gpt-oss-120b and models from OpenAI, Google and xAI among them; its results are rows of the capability list, not columns of this matrix ([model capabilities](../../architecture/model-capabilities.md#the-list)).
|
||||
|
||||
| Rung | Models | How |
|
||||
|---|---|---|
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
status: proposed
|
||||
status: in-progress
|
||||
code:
|
||||
- document_skills/
|
||||
- surfsense_local/backend/shared/migrations.py
|
||||
|
|
@ -37,7 +37,7 @@ code:
|
|||
|
||||
This is the entry point of the file-agent proposal. It extends the [agent proposal](../agent/README.md) and changes some of its locked decisions. The six stream files carry the design and its verified facts; this page resolves where they disagree, orders their phases, and lists what is left to the maintainer. The stream files were reconciled with this page on 3 Oct 2026. Facts were checked against `dev_mod` at `0847e12f7` on 3 Oct 2026. The workspace question went to a design panel of three proposals, and both judges chose P1 (48 points, against 38 to 41 for P2 and 40 for P3).
|
||||
|
||||
**The first slice comes before the milestones.** [07, create and edit](07-create-and-edit-mvp.md) is built first, on `dev_mod`, which syncs with `dev` every few hours: the agent creates a Word or PDF file in chat and refines it turn by turn through a document script it keeps. During the slice, Studio's buttons stay as they are today. It pulls reduced parts of M3 (versions) and M5 (the agent) forward, and it follows [ADR 0039](../../adr/0039-document-scripts-run-without-approval.md): document scripts run without approval in SurfSense's script runner, and the agent has no shell. Studio's Word and PDF change after the slice, into two paths ([07](07-create-and-edit-mvp.md#after-the-slice)): strong models keep today's code-written documents, which are proven, and small models write Markdown that committed builders render. Each path has its own one-call Refine, and a rule decides which models count as strong. No reliable signal for that rule exists yet; [05](05-model-ladder-and-evals.md)'s measured capability is where it will come from. Where 07 disagrees with the milestones below or with [02](02-skills-and-engines.md)'s Studio builders, 07 holds, including its two paths for Studio's Office formats after the slice, and the milestones are revised once the slice ships.
|
||||
**The first slice comes before the milestones.** [07, create and edit](07-create-and-edit-mvp.md) is built first, on `dev_mod`, which syncs with `dev` every few hours: the agent creates a Word or PDF file in chat and refines it turn by turn through a document script it keeps. During the slice, Studio's buttons stay as they are today. It pulls reduced parts of M3 (versions) and M5 (the agent) forward, and it follows [ADR 0039](../../adr/0039-document-scripts-run-without-approval.md): document scripts run without approval in SurfSense's script runner, and the agent has no shell. Studio's Word and PDF change after the slice, into two paths ([07](07-create-and-edit-mvp.md#after-the-slice)): strong models keep today's code-written documents, which are proven, and small models write Markdown that committed builders render. Each path has its own one-call Refine, and a rule decides which models count as strong. No reliable signal for that rule exists yet; [05](05-model-ladder-and-evals.md)'s measured capability is where it will come from. Where 07 disagrees with the milestones below or with [02](02-skills-and-engines.md)'s Studio builders, 07 holds, including its two paths for Studio's Office formats after the slice, and the milestones are revised once the slice ships. The slice shipped in 2.2.0 on 9 Oct 2026, together with revised copies, Office support, PDF tools and data analysis; the milestones below are not revised yet.
|
||||
|
||||
**First ladder results, 4 Oct 2026.** The slice's eight live cases pass on Claude Opus 5.5 and Sonnet 5.5. Haiku 4.5 passes five: in the other three it skipped the page previews the skill asks it to check. Through OpenRouter, Kimi K3 and Qwen3.8-27B pass all eight cases. Text-only GLM-5.3 and DeepSeek V4 Pro pass all but the case that needs image input. Smaller open models (Qwen3.6-35B-A3B, Gemma 4 31B, Ministral 14B, Qwen3.5-9B) fall short. The matrix, the causes and the columns still to run are in [08](08-model-ladder-results.md).
|
||||
|
||||
|
|
@ -51,7 +51,7 @@ This is the entry point of the file-agent proposal. It extends the [agent propos
|
|||
|
||||
## The answer to the workspace question
|
||||
|
||||
Preload the mode from the model, and add no workspace types. The model is app-wide, so a workspace type could not carry a capability; a type would freeze today's limits into user data and ask a question new users cannot answer. Every benefit a type offered (folder roles, pinned jobs, a local-only policy, a review state) fits one workspace. `capability_of()` reads a reviewed, measured profile and decides engines, job levels, formats, edit options and skills. The interface never says "mode": each item carries one of four verdicts (Best on this model, Tested, Not measured, Not on this model) with its reason. A verdict only ever reflects eval rows and live gates; nothing else, such as a pending legal review, is shown as one. When a model allows both engines, a new thread offers "Answer" or "Work on files", so a 4B user never sees a locked control. The full answer, with the grafts taken from the two losing proposals, is in [06](06-product-shape.md#the-answer-to-the-maintainers-question).
|
||||
Preload the mode from the model, and add no workspace types. The model is app-wide, so a workspace type could not carry a capability; a type would freeze today's limits into user data and ask a question new users cannot answer. Every benefit a type offered (folder roles, pinned jobs, a local-only policy, a review state) fits one workspace. `capability_of()` reads a reviewed, measured profile and decides engines, job levels, formats, edit options and skills. The interface never says "mode": each item carries one of four verdicts (Best on this model, Tested, Not measured, Not on this model) with its reason. A verdict only ever reflects eval rows and live gates; nothing else, such as a pending legal review, is shown as one. When a model allows both engines, a new thread offers "Answer" or "Work on files", so a 4B user never sees a locked control. The full answer, with the grafts taken from the two losing proposals, is in [06](06-product-shape.md#the-answer-to-the-maintainers-question). What 2.2.0 shipped differs: the composer offers Basic (Q&A) and Agentic for each new chat, every model may pick Agentic unless a technical gate keeps it out, and a measured level sets only the default ([model capabilities](../../architecture/model-capabilities.md#the-modes)).
|
||||
|
||||
## Workstreams
|
||||
|
||||
|
|
@ -72,7 +72,7 @@ Preload the mode from the model, and add no workspace types. The model is app-wi
|
|||
|---|---|---|---|
|
||||
| Engines | Two: opencode for a model on a tested list, fixed workflows for every other model | Answers use the one-call chat engine on every model; a job runs as a job thread (opencode running a per-job agent that carries the job's skill) or a workflow; free-form file work runs on the free agent; the capability sets each level | Small models do single schema steps well and fail as agents ([05](05-model-ladder-and-evals.md#7-the-small-model-tier)) |
|
||||
| Who gets opencode | Models on the tested list, which starts empty | Models whose measured capability grants it, plus unmeasured models the user confirms once, behind live gates | A boolean list cannot say how each job runs ([05](05-model-ladder-and-evals.md#4-the-capability-profile)) |
|
||||
| When installers carry opencode | Once a model passes the agent test ([`enabled.mjs`](../../../surfsense_local/electron/scripts/opencode/enabled.mjs): `ENABLED_BY_DEFAULT = false`) | Since 7 Oct 2026 (`ENABLED_BY_DEFAULT = true`): the maintainer turned it on ahead of M5's `agent_smoke` and `free_agent_tasks` rows. A notarized macOS build carrying opencode is a macOS release gate; if notarization is still blocked, the agent ships on Windows and Linux first and the docs say so | No agent test exists to flip the switch ([agent](../../architecture/agent.md)); 05's rows replace it |
|
||||
| When installers carry opencode | Once a model passes the agent test ([`enabled.mjs`](../../../surfsense_local/electron/scripts/opencode/enabled.mjs): `ENABLED_BY_DEFAULT = false`) | Since 7 Oct 2026 (`ENABLED_BY_DEFAULT = true`): the maintainer turned it on ahead of M5's `agent_smoke` and `free_agent_tasks` rows. v2.2.0, on 9 Oct 2026, is the first release that carries it, on all three platforms, notarized on macOS and signed on Windows | No agent test exists to flip the switch ([agent](../../architecture/agent.md)); 05's rows replace it |
|
||||
| Where the agent appears | The thread gets its engine from the selected model; "a quick question also goes through the agent" | The model decides which engines a thread may use; when both are allowed, the user picks Answer or Work on files at creation. A thread keeps its engine | Without prompt caching, every quick question would bill opencode's fixed prompt and tool definitions: an estimated 3–5K tokens in SurfSense's config (6–8K for stock opencode), to be measured from `usage.input_tokens` in M4 ([06](06-product-shape.md#threads-answer-and-work-on-files)) |
|
||||
| Models | One SurfSense endpoint for every model | It stands, with two wires: chat completions, and an Anthropic Messages passthrough that refuses server tools | The compatibility layer drops caching, schemas and thinking ([05](05-model-ladder-and-evals.md#2-serving-claude-to-opencode)) |
|
||||
| Model-written code | The agent asks before every shell command | Decided in [ADR 0039](../../adr/0039-document-scripts-run-without-approval.md): the agent's shell is denied for every model, and model-written document scripts run without approval in SurfSense's script runner, kept with the version they made. Studio's Office formats move onto the runner after [07](07-create-and-edit-mvp.md)'s slice. Studio's Word and PDF then take two paths ([07](07-create-and-edit-mvp.md#after-the-slice)): strong models keep code-written documents, and small models write Markdown that committed builders render | Approval dialogs for every command stop document work, and a bash allow-list is no boundary ([02](02-skills-and-engines.md#10-security)); code-written documents are proven on strong models; free users must not lose a format they have today |
|
||||
|
|
@ -91,7 +91,7 @@ Each milestone ships on its own as a 2.x point release; exit criteria use the ev
|
|||
|
||||
**Durations are rough**, for one maintainer working with agents, and are replaced by measured velocity after M1. Along the longest chain, M0 → M1 → M4 → M5 → M6, the redline job lands about 14–20 weeks after M0 starts, between mid-January and late February 2027. The questionnaire (M7) follows within about eight weeks of M6, so between February and April 2027. Both dates are rough.
|
||||
|
||||
**Every milestone's exit also includes** one how-to page per new feature in [`content/docs/v2`](../../../surfsense_web/content/docs/v2/index.mdx), which today holds only a "Docs in progress" placeholder; a changelog entry and an announcement; the macOS status on [/downloads](../../../surfsense_web/app/\(home\)/downloads/page.tsx); and, at a milestone that adds a network destination, the [privacy page](../../../surfsense_web/app/\(home\)/privacy/page.tsx)'s list of destinations (section 3.2 lists three kinds today). The privacy page is also reviewed in M0.
|
||||
**Every milestone's exit also includes** one how-to page per new feature in [`content/docs/v2`](../../../surfsense_web/content/docs/v2/index.mdx), which today holds only a "Docs in progress" placeholder; a changelog entry and an announcement; and, at a milestone that adds a network destination, the [privacy page](../../../surfsense_web/app/\(home\)/privacy/page.tsx)'s list of destinations (section 3.2 lists three kinds today). The privacy page is also reviewed in M0.
|
||||
|
||||
| | Name | Depends on | Rough |
|
||||
|---|---|---|---|
|
||||
|
|
@ -166,14 +166,14 @@ Each milestone ships on its own as a 2.x point release; exit criteria use the ev
|
|||
### M5: The agent, on by default
|
||||
|
||||
- **Goal:** an agent scoped like chat, whose files become versions, in the installers.
|
||||
- **User sees:** "Answer / Work on files" on Claude; agent threads that read only the ticked sources; files the agent writes appear as artifact versions; the study-pack job; the agent in the Windows and Linux installers, and on macOS once a notarized build carries it.
|
||||
- **User sees:** "Answer / Work on files" on Claude; agent threads that read only the ticked sources; files the agent writes appear as artifact versions; the study-pack job; the agent in every installer, which 2.2.0 already shipped.
|
||||
- **Contains:** 01 phases 3a and 3b; 03 phase 3 (the outputs sweep); 06 S3, J1 and J2; 05 P4b (rows for Claude, the `free_agent_tasks` family, and the OpenAI and Gemini columns). `ENABLED_BY_DEFAULT` was flipped to true earlier, on 7 Oct 2026.
|
||||
- **Exit:**
|
||||
- Two agent threads keep separate views, and `create_artifact` refuses an unticked id; `agent_smoke` is 3 of 3 on Sonnet with per-thread folders; a file the agent writes in `outputs/` becomes a version.
|
||||
- `free_agent_tasks` (at least 10 cases) passes its gate on Sonnet. Each failed frontier case is labelled skill, tool, prompt or model, and fixed before lower rungs run it. Smoke and single-call columns for one OpenAI and one Gemini frontier model on the OpenAI-compatible wire are committed (05 P4b).
|
||||
- [agent.md](../../architecture/agent.md)'s "agent test" points to 05's `agent_smoke` and the capability rule. 04's size gate counts opencode and ripgrep (about +62 MB compressed, measured in 04), which installers carry since 7 Oct 2026.
|
||||
- Idle and busy private bytes per process are measured on a 16 GB Windows laptop with integrated graphics and recorded against the memory budget (Small things).
|
||||
- **Depends on:** M2, M3 and M4. A notarized macOS build carrying opencode gates the macOS release, not this milestone.
|
||||
- **Depends on:** M2, M3 and M4.
|
||||
- **Rough:** 3–4 weeks.
|
||||
|
||||
### M6: Revised copies and the redline job
|
||||
|
|
@ -209,7 +209,7 @@ Each milestone ships on its own as a 2.x point release; exit criteria use the ev
|
|||
- **Goal and user sees:** Link a folder on disk, indexed in place and watched, with root states (unavailable, permission denied, cloud-only, mass vanish), a root-level state when most of a root is OneDrive online-only files, and Office support for legacy formats.
|
||||
- **Contains:** 01 phases 4a and 4b, with the OneDrive state; 04 RT3a, RT3b (shown as "Office support") and RT6. Creating the pack-hosting repository, its publisher App and its protected environment is an explicit RT3a task, owned by the maintainer as the GitHub organization's admin.
|
||||
- **Exit:** 01's phase 4a platform tests pass from packaged apps on all three platforms, plus a packaged-app test on a Windows VM with OneDrive Known Folder Move; the dehydrated share of a real Documents folder is measured; an unavailable root keeps every document; a 600-file vanish is held; with Office support, `.doc`, `.xls` and `.ppt` are read, and without it the root header counts them; the privacy page lists `release-assets.githubusercontent.com` for packs.
|
||||
- **Depends on:** M2. It runs in parallel with M5–M7. ADR 0041 waits on TDF's answer, and the macOS packaged test needs a notarized build (outside dependencies below).
|
||||
- **Depends on:** M2. It runs in parallel with M5–M7. ADR 0041 waits on TDF's answer (outside dependencies below); the macOS packaged test needs a notarized build, which a dry run of the release workflow makes.
|
||||
- **Rough:** 4–6 weeks.
|
||||
|
||||
### M9: Down the ladder
|
||||
|
|
@ -232,7 +232,6 @@ Chase dates are proposed; each owner confirms or moves theirs.
|
|||
|---|---|---|---|
|
||||
| Haiku 4.5's retirement, "not sooner than" 2026-10-15 | the M1 gate's baseline | the maintainer, who runs the re-run | re-run committed by 2026-10-14 |
|
||||
| A clean-room spec author, a separate implementer, and a reviewer who never opened `original_skills/` | M1 | the maintainer | 2026-10-09 |
|
||||
| The Apple Developer agreement | every macOS release, the agent on macOS (M5), 01's packaged macOS test (M8) | the Apple account holder | 2026-10-09 |
|
||||
| A lawyer: first the provenance record and schema terms (02 phase 0L), then the starter playbooks | selling to legal and security buyers; public job pages, SEO pages and legal marketing | the maintainer | engaged by 2026-10-16; records sent at M1's exit |
|
||||
| TDF's answer on a trimmed LibreOffice and its source mirror | ADR 0041's acceptance, RT3b (M8) | the maintainer | asked by 2026-10-16, chased 2026-11-06 |
|
||||
| A 16 GB Windows laptop with integrated graphics | the memory budget at M5's and M6's exits | the maintainer | 2026-11-13 |
|
||||
|
|
@ -249,7 +248,7 @@ Numbers are proposed in this order from the current last ADR, [0039](../../adr/0
|
|||
| **0041** The running app may download pinned upstream runtime binaries after per-host consent | packs unaltered except for deleted files, pinned in the app, never SurfSense's own code; acceptance waits on TDF's answer (04's open questions) | 04 |
|
||||
| **0042** Third-party skill material enters only through a provenance record | the licence gate binds contributors and packs | 02 |
|
||||
| **0043** Claude goes through Anthropic's Messages API | amends ADR 0015: a connection's wire follows its host and `auth_kind` | 05 |
|
||||
| **0044** What a model may do comes from a reviewed, measured capability manifest | extends ADR 0014. One workspace kind. The tier is prompt shape only. An unmeasured model keeps today's behaviour. A verdict reflects eval rows and live gates only, never a release policy such as a legal hold. The license never changes a verdict, and an organization policy may only lower one | 05 and 06, merged into one |
|
||||
| **0044** What a model may do comes from a reviewed, measured capability manifest | extends ADR 0014. One workspace kind. The tier is prompt shape only. An unmeasured model keeps today's behaviour. A verdict reflects eval rows and live gates only, never a release policy such as a legal hold. The license never changes a verdict, and an organization policy may only lower one. What #2207 built differs: a measured level sets only the default mode ([#2220](https://github.com/MODSetter/SurfSense/issues/2220)) | 05 and 06, merged into one |
|
||||
| **0045** Retrieval scope is resolved on the server and applied inside both legs | the KNN prefilter | 01 |
|
||||
| **0046** SurfSense never loses a user's sources by surprise | Trash; read-only linked roots; no delete on unavailability; a mass vanish asks; placeholders never read; a database snapshot before every migration | 01 |
|
||||
| **0047** Premium plugins are source-available and delivered by the license server | premium plugins, such as the scraper client, live under the Business Source License 1.1 in `surfsense_backend/app/proprietary/`; the license server hands them only to a valid license; the open-source installer never carries them; the app, the engines, the format skills and all job content (job skills, playbooks, questionnaire mappings, RFP templates, eval cases) stay Apache-2.0 in `surfsense_local/`, and job packs installed through the plugin system stay open source. Supersedes [ADR 0025](../../adr/0025-scraper-client-as-paid-plugin.md)'s "paid plugin code is Apache-2.0" premise and the [plugins proposal](../plugins/README.md)'s "every plugin is Apache-2.0" for paid plugins | this page and the [strategy](../../../plans/community-local/file-agent-strategy.md) |
|
||||
|
|
@ -332,7 +331,7 @@ Each names the stream and phase that owns it.
|
|||
| Internal comments reach the counterparty | the external download is the default and removes them, with a leak scan in the report |
|
||||
| The finished stack exhausts a 16 GB laptop | the memory budget at M5's and M6's exits; one place for the encoder; ingest yields to a streaming turn |
|
||||
| Most of a linked Documents folder is OneDrive online-only | a root-level state with the fix; onboarding copy; SurfSense still never triggers a download |
|
||||
| macOS cannot carry the agent | Windows and Linux first, said on /downloads; a notarized build with opencode gates macOS |
|
||||
| Apple's developer agreement lapses and notarization stops, as it did on 3 Oct 2026 | the account holder accepts it again, which needs no code; 04's RT0 runs the credentials check on dry runs and weekly, so a lapse shows before a release |
|
||||
| opencode 1.x stops being maintained, or 2.x diverges | the pin; acceptance on 1.18.34; an adapter and a Goose spike on 02 phase 7's trigger |
|
||||
| An injected document asks for a silent change or exfiltration | engines write tracked changes only; no network imports; the reminder strip; server tools refused; scripts sandboxed |
|
||||
| A user's API bill surprises them | the ledger, estimates above $0.25, and a cap that answers 403 so it is never retried |
|
||||
|
|
@ -344,8 +343,8 @@ Each names the stream and phase that owns it.
|
|||
2. **When do you engage a lawyer, and for what?** *Recommended:* now, in parallel with M1's clean-room work, scoped to `provenance.json`, the scan report, the process record and the schema publishers' terms. Add the starter playbooks next.
|
||||
3. **Does the redline job need the playbooks' lawyer review before release?** *Recommended:* no. The job is offered on a model once its eval row passes the job gate, and no verdict stands in for a legal hold. Until a lawyer reviews them, each starter playbook carries "Starter playbook — not reviewed by a lawyer; not legal advice" on the job card, in the issues list and in the exported issues file; a user's own playbook carries no label. Public job pages, SEO pages and legal marketing wait for the review.
|
||||
4. **What eval budget, and which account?** *Recommended:* a dedicated Anthropic workspace under the organization, about $650 per full sweep, a $300 monthly cap outside sweeps, and the `model-evals` environment reviewed by you.
|
||||
5. **Who provides the reference machines?** *Recommended:* a 32 GB Apple-silicon Mac for the 27–35B rows, once the Apple agreement is current, and a 16 GB Windows laptop with integrated graphics for the memory budget. Until then, 27–35B rows are screened only, and community `matrix` submissions are the other route.
|
||||
6. **Who accepts the Apple Developer agreement, and when?** *Recommended:* the account holder, by 2026-10-09. It blocks every macOS release, the agent on macOS and 01's packaged privacy test.
|
||||
5. **Who provides the reference machines?** *Recommended:* a 32 GB Apple-silicon Mac for the 27–35B rows, and a 16 GB Windows laptop with integrated graphics for the memory budget. Until then, 27–35B rows are screened only, and community `matrix` submissions are the other route.
|
||||
6. ~~**Who accepts the Apple Developer agreement, and when?**~~ Answered: the account holder accepted it by 4 Oct 2026, when the 2.1.0 dry run's re-run and v2.1.0's tag build were notarized; v2.2.0's was notarized with opencode inside.
|
||||
7. **On routes that cache prompts, should a new thread default to Answer or to Work on files?** *Recommended:* default to Answer until M4 measures first-answer latency and cost on Sonnet. Switch to 05's rule only if Work on files is no slower for a one-line question.
|
||||
8. **May the LibreOffice download be trimmed and re-hosted, and must its full source be mirrored?** *Recommended:* ask TDF now, well before RT3b in M8, mirror the exact source archive beside each pack, and name it "Office support (LibreOffice)".
|
||||
9. **Should SurfSense ever trigger a cloud download of a placeholder?** *Recommended:* no. The user makes a file or a folder available offline in their sync client, guided by the root's online-only state, and the next reconcile picks it up.
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@ This extends the local catalog ([`catalog.md`](../architecture/local-models/cata
|
|||
| Podcast gate | `requires_voice`, "Needs a voice model" | `audio_gen` among the format's required model types, "Needs an audio model" |
|
||||
| Voices | a list hard-coded in `providers/kokoro/provider.py` | each model's roster, committed in the manifest |
|
||||
|
||||
The installer loses the 340 MB of Kokoro weights and three Python packages, and gains the sidecar. A podcast needs one download first, so the root README's two lines that say the podcast voice ships inside the installer change with the step that removes it.
|
||||
The installer loses the 340 MB of Kokoro weights and three Python packages, and gains the sidecar. A podcast needs one download first, so the root README's Podcast row, "voiced on your machine by default", and its translations are re-checked with the step that removes it; the lines that said the voice ships inside the installer went in the 9 Oct 2026 rewrite (#2212).
|
||||
|
||||
## Why audio.cpp
|
||||
|
||||
|
|
|
|||
|
|
@ -152,7 +152,7 @@ A VAE comes from Comfy-Org's ungated copy when the vendor's repo is gated, as BF
|
|||
|
||||
### One model, for the job that needs it
|
||||
|
||||
- **The runtime route answers from Studio's jobs.** `GET /llm/image/local/runtime` answers with the model set for the slot that the oldest running sd.cpp job needs, whether `image_gen`, `image_edit` or `video_gen`. With none running, it keeps the last one for 5 minutes after the last job ended, and then answers none. Electron's existing poll starts, restarts or stops `sd-server` to match ([`index.ts`](../../surfsense_local/electron/src/main/index.ts)).
|
||||
- **The runtime route answers from Studio's jobs.** `GET /llm/image/local/runtime` answers with the model set for the slot that the oldest running sd.cpp job needs, whether `image_gen`, `image_edit` or `video_gen`. With none running, it keeps the last one for 5 minutes after the last job ended, and then answers none. Electron's existing poll starts, restarts or stops `sd-server` to match ([`watch-image-model.ts`](../../surfsense_local/electron/src/main/sidecars/watch-image-model.ts)).
|
||||
- **Jobs that need the same model share it**; sd-server runs requests one at a time. A job that needs another model waits until it is the oldest. Studio runs four jobs at once ([ADR 0008](../adr/0008-two-job-queues.md)), so without this two jobs could each hold a model.
|
||||
- **A job checks the model before it posts.** `GET /sdapi/v1/sd-models` names the file sd-server was launched with, since `/v1/models` always answers `sd-cpp-local`. There is no health route; `GET /` answers once the server is listening.
|
||||
- **5 minutes** is the same backstop audio uses, and Ollama's default; SwarmUI clears VRAM after 10. It keeps a second image or an edit right after the first from reloading.
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ A plugin is published when it changed since the last published release, as [`../
|
|||
|
||||
### The release, step by step
|
||||
|
||||
It follows [`RELEASE.md`](../../../../../surfsense_local/RELEASE.md), which this work edits in two places. It makes releasing from `main` the rule: bump on `dev`, merge `dev` into `main` by pull request, tag `main`. Today it also allows tagging `dev`, which would skip the plan comment. And after publishing the draft, the maintainer checks that `plugins-go-live.yml` went green, and reruns it if not, the way `legacy-update-bridge` must be green before undrafting.
|
||||
It follows [`RELEASE.md`](../../../../../surfsense_local/RELEASE.md), which already releases from `main`: bump on `dev`, merge `dev` into `main` by pull request, tag the merge. This work edits it in one place: after publishing the draft, the maintainer checks that `plugins-go-live.yml` went green, and reruns it if not, the way `legacy-update-bridge` must be green before undrafting.
|
||||
|
||||
`upload`, `go-live` and `withdraw` share one `concurrency` group, `plugins-publishing`, and never cancel one another, so the live catalog is only ever written by one of them at a time. Without it, a `withdraw` finishing after a `go-live` would put an older release's catalog back on top.
|
||||
|
||||
|
|
|
|||
|
|
@ -13,6 +13,8 @@ code:
|
|||
|
||||
> Clicking a source opens its original file in a panel that slides out beside the sidebar, when the app has a viewer for that file type. PDF is the first. Every other type opens in the operating system's app, as every type does today. Each viewer added later moves one more type into the panel.
|
||||
|
||||
**Status, 11 Oct 2026:** step 1 is built, with PDF the only source viewer ([documents](../architecture/documents.md#the-original-file)). It departs from this page twice: the preview shows in place of the left sidebar, not in a second rail, opening at 560 px or at the sidebar's own width if that is wider, and since [#2195](https://github.com/MODSetter/SurfSense/pull/2195) its edge can be dragged, up to 800 px. Steps 2 and 3 are not built.
|
||||
|
||||
Today a source's title and its **Open** item both hand the file to the operating system ([`documents.md`](../architecture/documents.md), The original file). The app already renders PDF, DOCX, XLSX, PPTX, HTML and images, but only for Studio artifacts ([`studio.md`](../architecture/studio.md), Viewers). The idea and its duplicate check are in [#2077](https://github.com/MODSetter/SurfSense/issues/2077).
|
||||
|
||||
## What changes
|
||||
|
|
@ -69,5 +71,4 @@ The change that completes the first step updates [`documents.md`](../architectur
|
|||
|
||||
- Any change to citations: a citation keeps opening the citation panel on the right. Opening the preview at a cited page would also need a mapping, since chunks carry line ranges of the extracted markdown, not PDF pages.
|
||||
- Any change to the artifact panel, the Studio panel or Studio's viewers, including moving them onto the shared registry.
|
||||
- A preview wider than 400 px on a larger window, or a resizable rail. `SlideRail` takes one fixed width.
|
||||
- Previewing notes. A note has no file behind it.
|
||||
|
|
|
|||
|
|
@ -84,10 +84,10 @@ measured on any model yet
|
|||
multi-turn 22.1% for Qwen3-4B-Instruct-2507,
|
||||
[leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html)).
|
||||
- **Anything about timing against Cowork's change.** It took effect on 6 Oct
|
||||
2026, and SurfSense's agent reaches installers only with the first release
|
||||
built after 7 Oct 2026, when `ENABLED_BY_DEFAULT` turned true in
|
||||
[`enabled.mjs`](../../surfsense_local/electron/scripts/opencode/enabled.mjs).
|
||||
Until that release is out there is nothing to point a Cowork user to.
|
||||
2026, and SurfSense's agent reached installers with 2.2.0 on 9 Oct 2026,
|
||||
after `ENABLED_BY_DEFAULT` turned true in
|
||||
[`enabled.mjs`](../../surfsense_local/electron/scripts/opencode/enabled.mjs)
|
||||
on 7 Oct.
|
||||
|
||||
## Who buys first, and which job ships first
|
||||
|
||||
|
|
@ -191,7 +191,7 @@ the app's download, consent and install path, all before the first premium
|
|||
plugin ships.
|
||||
|
||||
**Gaps in the paid offer today.** No plugin exists, the pricing page still says
|
||||
"No plugin ships in version 2.0" while the app is 2.1.0
|
||||
"No plugin ships in version 2.0" while the app is 2.2.0
|
||||
([`package.json`](../../surfsense_local/electron/package.json), line 6), and the
|
||||
Enterprise card promises a mirror, central policy and an egress audit log that
|
||||
are not built (I8 gap 2). Until the scraper plugin ships, priority support is
|
||||
|
|
@ -323,14 +323,13 @@ Two rules make the wording hold:
|
|||
|
||||
| Surface | Today | Change | Gate |
|
||||
|---|---|---|---|
|
||||
| [README](../../README.md) lead | "The air-gapped, open-source NotebookLM alternative. Turn documents you can't upload into briefings, decks, reports, study guides and podcasts" | Lead with the edit: your folder in, your own file back with tracked changes, on your machine, with the model you choose. Keep NotebookLM in the comparison table, updated for its cloud computer and Office output | the first job passing on one model; until then only the comparison table changes |
|
||||
| README comparison table | NotebookLM rows on sources, models, licence | add rows: local folders, edits the user's file with tracked changes, model choice, where the agent runs; concede media breadth | none: these are facts about Gemini Notebook today |
|
||||
| [README](../../README.md) lead | since 9 Oct 2026 ([#2212](https://github.com/MODSetter/SurfSense/pull/2212)): "The air-gapped, open-source NotebookLM alternative." over "The AI agent that researches, transforms and edits your documents, on your machine.", a demo video of a contract redlined as tracked changes, and a table of what it reads, creates, edits as a copy and transforms per format; it names no model and no job | Lead with the edit: your folder in, your own file back with tracked changes, on your machine, with the model you choose. Keep NotebookLM in the comparison table, updated for its cloud computer and Office output | the first job passing on one model; the 9 Oct lead names editing, which the revise tool does today, before any job has shipped |
|
||||
| README comparison table | five rows against NotebookLM (now Gemini Notebook): offline use, whether documents leave the machine, editing files (added 9 Oct 2026), licence and models; no concession on media | add rows: local folders, edits the user's file with tracked changes, model choice, where the agent runs; concede media breadth | none: these are facts about Gemini Notebook today |
|
||||
| [`/private-ai-for-business`](../../surfsense_web/app/\(home\)/private-ai-for-business/page.tsx) | "Point it at a discovery folder…"; "Contract review runs against the executed copy sitting on your disk" | Read the copy against what has shipped. Linked folders are 01 phase 4a (M8) and the redline is 06 J3 (M6); until then the lines should describe uploads and Q&A | ship-dependent |
|
||||
| Every surface naming privacy | "Documents never leave your laptop" | add the qualifier: "unless you choose a cloud model" | none |
|
||||
| [Privacy page](../../surfsense_web/app/\(home\)/privacy/page.tsx) | its description says "your documents never leave your machine"; section 3.2 lists three kinds of destination: update checks to github.com, model downloads to registry.ollama.ai and huggingface.co, and a model provider the user configures | M0: the qualifier. Then each milestone that adds a destination names it: api.anthropic.com in M4, including `count_tokens` calls, which send the text being counted ([05](../../docs/proposals/file-agent/05-model-ladder-and-evals.md#1-the-native-anthropic-provider-chat-titles-studio)); release-assets.githubusercontent.com for packs in M8 ([04](../../docs/proposals/file-agent/04-runtime-and-packs.md) decisions 9 and 18) | the release that adds the destination |
|
||||
| [User docs](../../surfsense_web/content/docs/v2/index.mdx) | one placeholder page: "Its documentation is being written now" | one how-to page per new feature in `content/docs/v2` at each milestone's exit: folders (M2); versions and Refine (M3); the Claude key and spending caps (M4); Answer and Work on files (M5); revised copies and playbooks (M6); the questionnaire (M7); linking a folder and Office support (M8); a troubleshooting page that grows with each | each milestone's release |
|
||||
| [Changelog](../../surfsense_web/changelog/content/) and [announcements](../../surfsense_web/lib/announcements/announcements-data.ts) | the newest changelog entry is dated 16 Aug 2026; v2.0.0 was tagged on 18 Sep 2026 | a changelog entry and an announcement for every milestone release, starting with M0's 2.1.x point release | each release |
|
||||
| [Downloads](../../surfsense_web/app/\(home\)/downloads/page.tsx) | Windows, macOS and Linux panels, with no macOS status | a macOS status line: notarization blocked since 3 Oct 2026, and, if it still is in M5, the agent ships on Windows and Linux first | now (M0) |
|
||||
| [Pricing](../../surfsense_web/components/pricing/pricing-content.ts) | Individual sells scraper plugins; "No plugin ships in version 2.0" | correct the version; mark unbuilt Enterprise items as planned; keep the license to premium plugins, priority support and Enterprise, and price no job or job content | now (M0) |
|
||||
| Job pages (new) | none | one page per job, following the vendors that win organically with one page per artifact (08): "Fill a security questionnaire from your own policies", "Redline a contract against your playbook", "Answer an RFP from past proposals"; each names the document languages its job supports | a committed passing row on at least one model; for the redline page and any legal SEO page, also the lawyer's review of the starter playbooks |
|
||||
| Model page (new) | none | "Which model can do which job", generated from the matrix | the first committed sweep (05 P3a, M4) |
|
||||
|
|
@ -369,10 +368,10 @@ says which release carries each.
|
|||
|
||||
| # | Phase | Scope | Milestone | Depends on | Size |
|
||||
|---|---|---|---|---|---|
|
||||
| B0 | Honest copy now | the privacy qualifier on every surface and on the privacy page; landing lines matched to what has shipped; pricing version fix; Enterprise items marked planned; the macOS status on /downloads; a changelog entry and announcement for the 2.1.x point release | M0 | nothing | S |
|
||||
| B0 | Honest copy now | the privacy qualifier on every surface and on the privacy page; landing lines matched to what has shipped; pricing version fix; Enterprise items marked planned; a changelog entry and announcement for the 2.1.x point release | M0 | nothing | S |
|
||||
| B1 | Clean engines | the clean-room fix and no-regression gate (02 phase 0), with the provenance legal review (0L) in parallel; the baseline re-run at n=3 on Haiku 4.5 and on Sonnet 5.5 at low effort before 15 Oct 2026, the earliest date Haiku 4.5 may retire | M1 | a spec author and a separate implementer | L |
|
||||
| B2 | Frontier evidence | native Anthropic provider, spend ledger and caps, first paid sweep (05 P1, P2, P3a); the privacy page names api.anthropic.com; how-to pages for the Claude key and spending caps | M4 | B1 | M, about $150 |
|
||||
| B3 | Redline in the app | 02 phases 1, 1b, 3, 4 and 6; 03 phase 4; 06 J3; revised copies download without internal comments by default; the user's own playbook and "Make a playbook from this document"; workspace and playbook export; free, with the starter playbook labelled | M6, after the agent is on in installers (M5) | B1, B2; versions (M3); per-thread folders and `copy_original` (M5) | L |
|
||||
| B3 | Redline in the app | 02 phases 1, 1b, 3, 4 and 6; 03 phase 4; 06 J3; revised copies download without internal comments by default; the user's own playbook and "Make a playbook from this document"; workspace and playbook export; free, with the starter playbook labelled | M6, after M5; the agent is in installers since 2.2.0 | B1, B2; versions (M3); per-thread folders and `copy_original` (M5) | L |
|
||||
| B4 | Questionnaire | the xlsx engine (02 phase 5a), started right after M1; 06 J4, free, with its mappings and eval cases Apache-2.0 in the app. Open-source job packs through the plugin installer (02 phase 8, 06 P1) follow in M10, never paywalled | M7 | B1; 02 phase 1 and 03 phase 4 from M6; not the redline workflow | XL |
|
||||
| B5 | Pages | model page; questionnaire job page; redline page and legal SEO pages after the provenance legal review (02 phase 0L) and the lawyer's review of the starter playbooks | from M4 (model page) and M7 (questionnaire page) | committed rows from B2–B4 | M |
|
||||
| B6 | Enterprise | organization policy (06 E1); audit log and mirror when a customer's deployment needs them | M10 | an Enterprise conversation | M |
|
||||
|
|
@ -391,7 +390,7 @@ says which release carries each.
|
|||
| A competitor copies the open job content | SurfSense's playbooks, mappings or templates ship in another product | Apache-2.0 allows it; the lead rests on the app, the engines and the published eval rows; premium plugins stay under the Business Source License |
|
||||
| Frontier cost deters users | measured token medians well above these estimates | default to the workflow level where it passes; cheaper columns earn rows |
|
||||
| The 4B tier fails its own floors | 4B rows below 05 section 7 targets | the free promise narrows to Q&A and the formats that pass; the wording above already allows it |
|
||||
| macOS cannot ship | notarization failed on 3 Oct 2026 (Apple agreement expired; [04](../../docs/proposals/file-agent/04-runtime-and-packs.md#open-questions)) | Windows and Linux first, the agent included from M5; a notarized macOS build that carries opencode is a macOS release gate; the downloads page says so |
|
||||
| macOS cannot ship | notarization stops when Apple's developer agreement lapses, as on 3 Oct 2026 until the account holder accepted it; the v2.1.0 and v2.2.0 builds were notarized, 2.2.0's carrying opencode ([04](../../docs/proposals/file-agent/04-runtime-and-packs.md#signing-and-notarization)) | the account holder accepts it, which needs no code; Windows and Linux ship meanwhile |
|
||||
| License holders get nothing | scraper license mode not live when hosted tokens are purged on 18 Oct 2026 (I8 gap 1) | priority support is the license's deliverable until the scraper plugin ships through the license server, so consider extending affected licenses; what else a license includes is revisited after the video and real user feedback (What stays free and what is sold) |
|
||||
|
||||
**What would change the order of jobs:** a measured questionnaire engine in
|
||||
|
|
@ -403,8 +402,8 @@ be had soon. Either moves the questionnaire to ship first.
|
|||
Each with a recommended answer.
|
||||
|
||||
1. **Does the free promise cover the agent?** Yes. Decide it in the open now,
|
||||
because the FAQ is live, 06 already assumes it, and the agent is on in
|
||||
installers from M5.
|
||||
because the FAQ is live, 06 already assumes it, and the agent has been in
|
||||
installers since 2.2.0 (9 Oct 2026).
|
||||
2. **Where do premium plugins live?** Decided: under the Business Source
|
||||
License in `surfsense_backend/app/proprietary/`, served by the license
|
||||
server to a valid license, with `surfsense_backend` otherwise reduced to the
|
||||
|
|
@ -435,16 +434,18 @@ Each with a recommended answer.
|
|||
The README, 02 and 06 state the same rule.
|
||||
7. **Must a sandbox exist before selling to security buyers?** No, but two
|
||||
things must: job threads with bash denied, and Studio's `exec()` of
|
||||
model-written code inside the worker replaced. After the create-and-edit
|
||||
model-written code inside the worker replaced, which
|
||||
[#2184](https://github.com/MODSetter/SurfSense/pull/2184) did on 11 Oct
|
||||
2026. After the create-and-edit
|
||||
slice, Studio's Word and PDF take two paths
|
||||
([07](../../docs/proposals/file-agent/07-create-and-edit-mvp.md#after-the-slice)):
|
||||
small models write Markdown that committed builders render, and strong
|
||||
models keep writing a script, which runs in SurfSense's script runner, a
|
||||
separate process killed at its time limit
|
||||
([ADR 0039](../../docs/adr/0039-document-scripts-run-without-approval.md)).
|
||||
PowerPoint and Excel take the same two paths: a script on the runner for
|
||||
strong models, and for small models a builder from a flat spec once it
|
||||
scores at least the script path's baseline (02 phase B); until then a
|
||||
PowerPoint and Excel run their scripts on the runner for every model
|
||||
since #2184, and move to a builder from a flat spec for small models once
|
||||
it scores at least the script path's baseline (02 phase B); until then a
|
||||
format keeps its script, so free users lose nothing. No
|
||||
model-written code runs in a job on any model. The runner is not a sandbox:
|
||||
a script runs with the user's privileges, and ADR 0039 has this answer
|
||||
|
|
@ -466,6 +467,6 @@ Each with a recommended answer.
|
|||
11. **Should job pages deep-link into the app?** Not in v1: link to downloads
|
||||
and the job's setup instructions. A `surfsense://` handler is new attack
|
||||
surface ([06, open questions](../../docs/proposals/file-agent/06-product-shape.md#open-questions)).
|
||||
12. **Should SurfSense speak to Cowork's change on 6 Oct?** Not until the agent
|
||||
ships in installers (M5). Pull the keyword data, and write one factual
|
||||
comparison page only if demand exists.
|
||||
12. **Should SurfSense speak to Cowork's change on 6 Oct?** Only once the agent
|
||||
ships in installers, which 2.2.0 did on 9 Oct 2026. Pull the keyword data,
|
||||
and write one factual comparison page only if demand exists.
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ This is the demand document. The rest of the set:
|
|||
| [`03-international.md`](03-international.md) | 26 markets in 15 languages: market sizes, localisation tiers, per-language glossaries, hreflang plan |
|
||||
| [`04-competitors.md`](04-competitors.md) | domain authority, competitor footprints, keyword gaps, patterns to copy; the edtech set that owns the Studio SERPs |
|
||||
| [`05-serp-landscape.md`](05-serp-landscape.md) | what the results page looks like for the four priority terms and the study-guide head, AI Overview citations, PAA |
|
||||
| [`06-repo-readme.md`](06-repo-readme.md) | the repo as a ranking page: description, topics, and a paste-ready README draft in [`drafts/`](drafts/README.md) |
|
||||
| [`06-repo-readme.md`](06-repo-readme.md) | the repo as a ranking page: description, topics, the README's structure and how to change it; the 2.0.0 draft in [`drafts/`](drafts/README.md) is kept for history |
|
||||
| [`07-what-users-do.md`](07-what-users-do.md) | **what our own users actually ask for**, from production chat data — the audience check on everything in this file |
|
||||
| [`data/master-keywords.csv`](data/master-keywords.csv) | 565 keywords scored and clustered, the single list to work from |
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,13 @@
|
|||
# The repo: description, topics and README
|
||||
|
||||
The paste-ready README draft is [`drafts/README.md`](drafts/README.md). This file
|
||||
is why it says what it says, the shorter strings that go around it, and what has
|
||||
to be checked before it ships.
|
||||
The root [README](../../../README.md) is edited directly. It was rewritten short
|
||||
on 8 and 9 Oct 2026 (#2212), and [The README since 9 Oct 2026](#the-readme-since-9-oct-2026)
|
||||
says what it holds, why, and how to change it. The rest of this file is the
|
||||
reasoning behind the 2.0.0 README of 18 Sep 2026, whose paste-ready draft,
|
||||
[`drafts/README.md`](drafts/README.md), is kept for history. Its research on the
|
||||
description, topics, competitors and length still applies, and so does the rule
|
||||
that anything added replaces something; its section notes and its checklist
|
||||
describe that older file.
|
||||
|
||||
Why this is an SEO document at all: GitHub ranks **#1** for `open source
|
||||
notebooklm`, `local notebooklm`, `notebooklm mcp` and `notebooklm api`, and is in
|
||||
|
|
@ -12,6 +17,107 @@ The repo outranks anything the marketing site will achieve on those terms, so
|
|||
the description is a meta description and the README's first paragraph is what
|
||||
gets quoted into AI Overviews.
|
||||
|
||||
## The README since 9 Oct 2026
|
||||
|
||||
Rewritten short on 8 and 9 Oct 2026 (#2212), as 2.2.0 became the first release
|
||||
whose installers carry the agent. It is about 450 words under six H2s, at the
|
||||
short end of the [length table](#length-the-biggest-projects-have-the-shortest-readmes).
|
||||
In order:
|
||||
|
||||
1. **The header.** Badges, the banner and the H1, then the bold line **"The
|
||||
air-gapped, open-source NotebookLM alternative."**, which keeps the terms the
|
||||
repo ranks for, and the sub-line "The AI agent that researches, transforms
|
||||
and edits your documents, on your machine.", which leads with the edit, as
|
||||
the [file agent strategy](../file-agent-strategy.md#the-headline) asks. Then
|
||||
the link row, Trendshift and the language row.
|
||||
2. **The demo video** (below), one paragraph, the download table and the sunset
|
||||
callout, which goes on 18 Oct 2026 here and in every translation.
|
||||
3. **Quick start** in three steps: install and onboard, drop files or folders in
|
||||
the sources, ask the agent. A five-step version with screenshots was cut
|
||||
before its images were recorded.
|
||||
4. **What it does with each format.** One row per format: what the app answers
|
||||
from with citations, creates, edits as a copy and transforms, with three
|
||||
footnotes. It replaced "What SurfSense makes", the table of Studio outputs,
|
||||
because what the agent does to the user's own files is now the pitch.
|
||||
5. **How it compares**, against NotebookLM (now Gemini Notebook) alone: offline
|
||||
use, whether documents leave the machine, editing files, licence and models.
|
||||
"Everything stays on your machine", the comparisons with local chat apps and
|
||||
study tools, and the concession on audio and video overviews went; the
|
||||
privacy claims survive as the opening paragraph's last sentence and two rows.
|
||||
6. **Community**, **Star history** and **License**. License says Apache-2.0 and
|
||||
links `LICENSE`, which names the BSL 1.1 exception for
|
||||
`surfsense_backend/app/proprietary/`.
|
||||
|
||||
The repo's About field, read on 11 Oct 2026, follows the header rather than the
|
||||
[project description](#project-description) below: "Air-gapped,
|
||||
privacy-focused, open-source NotebookLM alternative. AI agent that researches,
|
||||
transforms, and edits your documents." and a Discord link. It has nine topics,
|
||||
not the sixteen below.
|
||||
|
||||
**Changing it.** Edit `README.md` first, then make the same change in every
|
||||
`README.<locale>.md`. Each follows the English section for section and keeps its
|
||||
own register and terms. In each one:
|
||||
|
||||
- name what the app shows with that language's own string in
|
||||
[`surfsense_local/frontend/translations/`](../../../surfsense_local/frontend/translations/)`<locale>.json`,
|
||||
such as Settings or Office support, rather than translating it afresh;
|
||||
- in Arabic, which the app does not translate, follow the Arabic with the
|
||||
English UI name in parentheses;
|
||||
- point the nav row at that file's own translated headings, such as
|
||||
`#schnellstart` in German.
|
||||
|
||||
The top comment of `README.md` says the same in brief.
|
||||
|
||||
**The format table** was checked cell by cell against `c5ff7938a` on `dev`
|
||||
(9 Oct 2026). v2.2.0 contains it; between the two, only the composer's mode
|
||||
menus (#2211) and the version bump changed the app, and neither touches a cell.
|
||||
Before each release, re-check every cell against what it ships
|
||||
([`RELEASE.md`](../../../surfsense_local/RELEASE.md), step 4), and change the
|
||||
translations with any cell that moves:
|
||||
|
||||
| Claim | Where it is true |
|
||||
|---|---|
|
||||
| Research: every listed format is read and cited; scanned pages and images through OCR, in Latin, Chinese and Japanese script | [`storage.py`](../../../surfsense_local/backend/modules/documents/storage.py) (the types accepted), [`parsing.py`](../../../surfsense_local/backend/worker/ingestion/parsing.py) (RapidOCR); the scripts OCR misreads are a [documents](../../../docs/architecture/documents.md#known-gaps) Known gap |
|
||||
| Excel: the agent reads formulas | [`read_workbook.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/read_workbook.py) |
|
||||
| Create Word, PDF, PowerPoint and Excel; Word and PowerPoint from the user's own file as a template; Excel with formulas and native charts | [`render_document.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/render_document.py) (`DOCUMENT_FORMATS`, `template_source_id`), the [`surfsense-documents`](../../../surfsense_local/backend/modules/agent/skills/surfsense-documents/SKILL.md) skill |
|
||||
| Create Markdown, a web page, images and infographics (with an image model), a podcast, mind maps, flashcards and quizzes | [`formats.py`](../../../surfsense_local/backend/modules/artifacts/formats.py); the podcast's default voice is in [studio](../../../docs/architecture/studio.md#formats) |
|
||||
| Edit Word body text as tracked changes and comments, Excel cell values and formulas, PowerPoint text on slides and notes, and delete or duplicate slides, always on a copy | [`revise_document.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/revise_document.py) and its [engines](../../../surfsense_local/backend/modules/artifacts/revised_copies/engines/) |
|
||||
| PDF: fill and flatten forms; stamp a watermark, page numbers, a header or a footer; merge, split, extract, rotate and reorder pages, each into a new PDF | [`pdf_form.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/pdf_form.py), [`pdf_stamp.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/pdf_stamp.py), [`pdf_pages.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/pdf_pages.py) |
|
||||
| Analyse CSV and Excel with pandas and chart the results | [`analyze_data.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/analyze_data.py) |
|
||||
| Place images in new documents | [`list_images.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/list_images.py) |
|
||||
| Word and PowerPoint to PDF with Office support, which is never in the installer | [`convert_document.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/convert_document.py), [office pack](../../../docs/architecture/office-pack.md) |
|
||||
|
||||
**The demo video** is a GitHub upload, not a file in the repo: a
|
||||
`github.com/user-attachments/assets/...` URL alone on its line, which GitHub
|
||||
plays inline. All eleven READMEs use the same one. To replace it, drag an MP4
|
||||
under 10 MB into any comment box in this repo, copy the `user-attachments` link
|
||||
GitHub writes, and swap the URL in every README.
|
||||
|
||||
The video, added on 9 Oct 2026, shows the agent redlining a city contract as a
|
||||
revised copy, with tracked changes and a comment on each change. It is step 4 of
|
||||
the Boston demo plan, `references/demo-data/Boston-demo-plan.md`, which git
|
||||
ignores along with its sources and the credits for them; the plan names Qwen3.8
|
||||
27B through OpenRouter for the recording.
|
||||
|
||||
- **Source:** `Boston-Standard-Contract-General-Conditions-CM11.docx`, a Word
|
||||
copy of the City of Boston's Standard Contract General Conditions (Form CM 11),
|
||||
in the plan's `Contracts/` folder beside the City's PDF of Forms CM 10 and
|
||||
CM 11 from content.boston.gov. Neither file is in the repo. To record again
|
||||
elsewhere, take that PDF and make a Word copy of the General Conditions with
|
||||
the same words; how the present copy was made is not written down.
|
||||
- **Query:** "We're a vendor about to sign Boston's standard contract. In
|
||||
Boston-Standard-Contract-General-Conditions-CM11.docx, propose changes as
|
||||
tracked changes: payment within 30 days of an approved invoice,
|
||||
indemnification in 7.3 limited to claims caused by our negligence, and a
|
||||
mutual right to terminate on 30 days' notice. Add a comment on each change
|
||||
explaining why."
|
||||
- **Rehearsal:** three runs on Qwen3.8 27B through OpenRouter on 6 Oct 2026.
|
||||
Each made one revised copy holding all three changes as tracked changes, with
|
||||
a comment on each, in 7 to 10 minutes and for $0.06 to $0.17 as OpenRouter
|
||||
billed. The rehearsal's own check, a scratch test never committed, failed one
|
||||
run on how it worded the mutual right ("the City or the Contractor"), which
|
||||
the copy does hold. The run folders are in `references/live-runs/rehearsal/`.
|
||||
|
||||
## Project description
|
||||
|
||||
GitHub's About field allows 350 characters, but Google truncates the snippet at
|
||||
|
|
@ -116,7 +222,7 @@ The two shortest files in the set belong to projects with 77k and 37k stars;
|
|||
the longest belongs to the one with 39k. gpt4all says what it is in 610 words.
|
||||
|
||||
The draft was **2,730 words across 14 H2s** when this was measured, longer than
|
||||
every repo in the category including the 152k-star one. It is now about 1,760
|
||||
every repo in the category including the 152k-star one. It shipped at about 1,760
|
||||
across 11, which sits between private-gpt and open-webui. Three cuts got it
|
||||
there, and they are the ones to repeat if it grows again:
|
||||
|
||||
|
|
@ -340,6 +446,8 @@ wherever the claim lives.
|
|||
server stack` section was removed for length, the sentence was kept inside
|
||||
`## Docs, roadmap and community`. Losing the section is fine; losing the
|
||||
sentence is a cross-workstream breach.
|
||||
The 9 Oct 2026 rewrite removed the self-hosting line that carried it
|
||||
(#2212), so the README no longer says it.
|
||||
- **Chat.** The draft briefly said "there is no box here to paste it into",
|
||||
which reads as though the app has no chat. It has: `modules/chat/` streams a
|
||||
cited answer per turn, and the Quick start has always said so four lines
|
||||
|
|
@ -378,7 +486,10 @@ RBAC or connector feature from the hosted product, all of which are sunset.
|
|||
|
||||
## Before it ships
|
||||
|
||||
- [ ] **Record a Studio demo GIF.** This is the highest-value missing asset,
|
||||
- [x] **Record a Studio demo GIF.** Done another way on 9 Oct 2026: a video of a
|
||||
contract redline, not a podcast GIF
|
||||
([The README since 9 Oct 2026](#the-readme-since-9-oct-2026)). The 17 Sep
|
||||
reasoning follows. This is the highest-value missing asset,
|
||||
because all three leaders put a visual above their feature list and we
|
||||
have none for the desktop app. `surfsense_local/` contains one image, an
|
||||
Electron icon. The hosted GIFs in `surfsense_web/public/homepage/` show
|
||||
|
|
@ -427,7 +538,7 @@ RBAC or connector feature from the hosted product, all of which are sunset.
|
|||
- [ ] **Fill the real links.** `/downloads`, `/docs`, `/pricing` and `/sunset`
|
||||
do not all exist yet (B5, B6); `/downloads` in particular must link a
|
||||
pinned tag, never `/releases/latest`, which stays on legacy v0.0.40.
|
||||
- [ ] **Rewrite the four localised READMEs before restoring the language
|
||||
- [x] **Rewrite the four localised READMEs before restoring the language
|
||||
switcher.** The switcher is commented out in the draft, not deleted, with
|
||||
the reason inline. `README.es.md`, `README.pt-BR.md`, `README.hi.md` and
|
||||
`README.zh-CN.md` all still open on the hosted positioning ("la
|
||||
|
|
@ -439,6 +550,9 @@ RBAC or connector feature from the hosted product, all of which are sunset.
|
|||
four markets, so either rewrite them cheaply from the English draft or
|
||||
leave the row commented; Japanese and German are the two localisations
|
||||
the research does fund, and neither has a README today.
|
||||
Done: the four were rewritten and the row restored on 21 Sep 2026, Arabic,
|
||||
German, French, Japanese, Korean and Russian followed on 22 Sep, and all
|
||||
ten follow the 9 Oct README (#2212).
|
||||
- [x] **Fill `REPLACE-ME-DATE` in the sunset callout** — 18 October 2026
|
||||
(T+30 from 18 Sep 2026 launch). Still **delete the callout at T+30**
|
||||
along with the "Importing from the hosted app" link.
|
||||
|
|
|
|||
|
|
@ -59,10 +59,6 @@ def check_links(path: Path) -> list[str]:
|
|||
for target in LINK.findall(line):
|
||||
if target.startswith(("http://", "https://", "#", "mailto:")):
|
||||
continue
|
||||
# Deliberate placeholders; 06-repo-readme.md wants them to fail
|
||||
# visibly in a browser, which is not this check's job.
|
||||
if "REPLACE-ME" in target:
|
||||
continue
|
||||
plain = re.sub(r"\\(.)", r"\1", target.split("#", 1)[0])
|
||||
if not (base / plain).resolve().exists():
|
||||
problems.append(f"{path.relative_to(ROOT)}:{lineno} dead link -> {target}")
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Desktop release (`vX.Y.Z`)
|
||||
|
||||
One number. Edit `VERSION`, copy it through the tree, tag the same number, push. GitHub’s **Latest** badge moves to each new 2.x release, which is the default — you do not pass a flag for it.
|
||||
One number. Edit `VERSION`, copy it through the tree, merge `dev` into `main`, and tag that merge with the same number. Publishing moves GitHub’s **Latest** badge to the new release; pass `--latest` anyway, so it does not rest on GitHub’s default.
|
||||
|
||||
Publishing a release makes it Latest, and the legacy 0.0.x desktop reads `latest*.yml` from whatever Latest names. The `legacy-update-bridge` job puts v0.0.40's manifests on every release so those clients keep landing on v0.0.40 and its `/sunset` redirect. **That job has to be green before you undraft**, or every legacy update check 404s.
|
||||
|
||||
|
|
@ -18,25 +18,39 @@ Publishing a release makes it Latest, and the legacy 0.0.x desktop reads `latest
|
|||
|
||||
## Ship it
|
||||
|
||||
From the repo root, on the branch you want to release (usually `dev` or `main` after merge). Replace `2.0.1` with this release, and `v2.0.0` with the previous 2.x you shipped.
|
||||
From the repo root, on `dev`. A release is `dev` merged into `main` by pull request, with the tag on that merge. Replace `2.3.0` with this release and `v2.2.0` with the previous 2.x you shipped.
|
||||
|
||||
Before you start, check both signing credentials are current. Apple refuses notarization with a 403 until the account holder accepts a new developer agreement, as on 3 Oct 2026, and only a tag's build checks for that before building ([#2224](https://github.com/MODSetter/SurfSense/issues/2224)). Windows signing needs the Azure client secret behind `AZURE_CLIENT_SECRET`, whose expiry Entra ID shows; only a tag's build uses it.
|
||||
|
||||
### 1. Write the new number
|
||||
|
||||
No `v` prefix.
|
||||
|
||||
```bash
|
||||
echo -n '2.0.1' > surfsense_local/VERSION
|
||||
echo -n '2.3.0' > surfsense_local/VERSION
|
||||
```
|
||||
|
||||
In Windows PowerShell, write it as ASCII. PowerShell 5.1's `>` and `-Encoding utf8` both add a byte-order mark, and the script then rejects the version it reads.
|
||||
|
||||
```powershell
|
||||
Set-Content surfsense_local/VERSION '2.3.0' -NoNewline -Encoding ascii
|
||||
```
|
||||
|
||||
### 2. Copy it through the tree
|
||||
|
||||
Writes electron, frontend, backend, the website download pin, and the license-email pin. On Windows run it through `bash`, not PowerShell. Do **not** reach for `scripts/bump-version.*` at the repo root — that is the hosted stack's own version line and it will not touch this app.
|
||||
Writes electron, frontend, backend, the website download pin, and the license-email pin, then refreshes `uv.lock` when `uv` is on the `PATH`. It needs bash and a Python that runs. Do **not** reach for `scripts/bump-version.*` at the repo root — that is the hosted stack's own version line and it will not touch this app.
|
||||
|
||||
```bash
|
||||
surfsense_local/scripts/bump-version.sh
|
||||
```
|
||||
|
||||
### 3. Commit
|
||||
On Windows, call Git Bash by its path. A plain `bash` in PowerShell is usually `C:\Windows\System32\bash.exe`, which starts WSL.
|
||||
|
||||
```powershell
|
||||
& "C:\Program Files\Git\bin\bash.exe" surfsense_local/scripts/bump-version.sh
|
||||
```
|
||||
|
||||
### 3. Commit and push to `dev`
|
||||
|
||||
Include whatever the script changed, including lockfiles.
|
||||
|
||||
|
|
@ -46,36 +60,70 @@ git add surfsense_local/VERSION \
|
|||
surfsense_local/frontend/package.json surfsense_local/electron/package.json \
|
||||
surfsense_web/lib/app-release.ts \
|
||||
surfsense_backend/app/license/release.py
|
||||
git commit -m "chore(local): release 2.0.1"
|
||||
git commit -m "chore(local): release 2.3.0"
|
||||
git push origin dev
|
||||
```
|
||||
|
||||
### 4. Push the commit
|
||||
### 4. Merge `dev` into `main`
|
||||
|
||||
The tag in the next step must point at a commit GitHub already has.
|
||||
Open the release pull request, read its checks, and merge it. Security Scan runs detect-secrets over every file the release changes since `main`, so it flags pinned sha256 hashes and test fixtures; read each finding ([#2225](https://github.com/MODSetter/SurfSense/issues/2225)).
|
||||
|
||||
Before merging, re-check every cell of the root README's format table against `dev` ([06-repo-readme](../plans/community-local/seo/06-repo-readme.md#the-readme-since-9-oct-2026)). A cell that changes, changes in every `README.<locale>.md` too, in a commit on `dev` before the merge.
|
||||
|
||||
```bash
|
||||
git push
|
||||
gh pr create --base main --head dev --title "[Local|Release] Release desktop 2.3.0"
|
||||
```
|
||||
|
||||
### 5. Open a draft release
|
||||
|
||||
Tag must be `v` + the `VERSION` file (CI fails if they disagree). Notes start at the **previous 2.x release** — the last one you shipped. `--draft` so CI can attach installers before anyone sees it. Pushing the tag starts Desktop Release.
|
||||
|
||||
Push the tag yourself rather than letting `gh` create it. `gh release create` puts a missing tag on the **default branch** head, which is the wrong commit whenever you release from `dev`.
|
||||
The merge also starts Build and Push Docker Images (`docker-build.yml`) on `main`, because the bump touches `surfsense_backend` and `surfsense_web`. It builds the hosted stack's images, not the desktop release. If its jobs fail on a registry outage, as 2.2.0's did when Docker Hub's token service answered 504, re-run them once the registry answers:
|
||||
|
||||
```bash
|
||||
git tag v2.0.1
|
||||
git push origin v2.0.1
|
||||
gh release create v2.0.1 --draft --generate-notes --notes-start-tag v2.0.0
|
||||
gh run list --workflow docker-build.yml --branch main --limit 1
|
||||
gh run rerun <run-id> --failed
|
||||
```
|
||||
|
||||
### 6. Publish after CI
|
||||
### 5. Tag the merge and open a draft release
|
||||
|
||||
Wait until Desktop Release is green on Mac, Windows, and Linux (installers and `stable-*.yml` on the draft) **and** `legacy-update-bridge` has attached `latest.yml`, `latest-mac.yml` and `latest-linux.yml`. Check all six manifests are there:
|
||||
Tag must be `v` + the `VERSION` file (CI fails if they disagree), on the commit the merge made on `main`. Pushing the tag starts Desktop Release. Push it yourself: `--verify-tag` makes `gh` stop when the remote lacks the tag, rather than create one on the default branch's head. Notes start at the **previous 2.x release** — the last one you shipped. `--draft` so CI can attach installers before anyone sees it.
|
||||
|
||||
```bash
|
||||
gh release view v2.0.1 --json assets --jq '.assets[].name' | grep -E '^(stable|latest)'
|
||||
gh release edit v2.0.1 --draft=false
|
||||
git fetch origin main
|
||||
git tag v2.3.0 "$(gh pr view <pr> --json mergeCommit --jq .mergeCommit.oid)"
|
||||
git push origin v2.3.0
|
||||
gh release create v2.3.0 --verify-tag --draft --generate-notes --notes-start-tag v2.2.0
|
||||
```
|
||||
|
||||
### 6. Watch the build
|
||||
|
||||
2.2.0's took 45 minutes. Wait until Desktop Release is green on Mac, Windows, and Linux (installers and `stable-*.yml` on the draft) **and** `legacy-update-bridge` has attached `latest.yml`, `latest-mac.yml` and `latest-linux.yml`. Check all six manifests are there:
|
||||
|
||||
```bash
|
||||
gh run list --workflow release-local.yml --limit 1
|
||||
gh run watch <run-id>
|
||||
gh release view v2.3.0 --json assets --jq '.assets[].name' | grep -E '^(stable|latest)'
|
||||
```
|
||||
|
||||
### 7. Add Known issues to the draft
|
||||
|
||||
The generated notes list pull requests. Above them, under `## Known issues`, write what a user must know before updating: what the first start needs, what going back to the previous version takes, and the bugs the release still has. 2.2.0's named the free disk its first start needs for the database snapshot, the hand restore of `~/.surfsense/backups/0023-0027.db` that going back to 2.1.0 takes, and the chats an interrupted import misses (#1958).
|
||||
|
||||
```bash
|
||||
gh release view v2.3.0 --json body --jq .body > notes.md
|
||||
# write "## Known issues" above "## What's Changed"
|
||||
gh release edit v2.3.0 --notes-file notes.md
|
||||
```
|
||||
|
||||
### 8. Try it over the previous version (optional)
|
||||
|
||||
The updater reads only published releases, so no installed app is offered the draft. Before publishing, install the draft's installer over the previous version on a machine that holds that version's library, and check the library opens and, when the release migrates the database, that the snapshot lands in `~/.surfsense/backups/`. Once published, Check now in Settings › About on a machine still on the previous version shows the update itself.
|
||||
|
||||
```bash
|
||||
gh release download v2.3.0 --pattern 'SurfSense-Setup.exe'
|
||||
```
|
||||
|
||||
### 9. Publish
|
||||
|
||||
```bash
|
||||
gh release edit v2.3.0 --draft=false --latest
|
||||
```
|
||||
|
||||
Do not undraft earlier. Undrafting makes this release Latest, and both updaters read their manifest off it the moment it is: a missing `stable-*.yml` strands 2.x clients, a missing `latest*.yml` strands every legacy 0.0.x client.
|
||||
|
|
@ -92,11 +140,18 @@ A `v*` tag starts `.github/workflows/release-local.yml`. electron-builder upload
|
|||
|
||||
## Dry run (no GitHub Release)
|
||||
|
||||
Actions → **Desktop Release** → version `0.0.0-test` (or any `X.Y.Z`) → publish **never**. Unsigned, not a tag, not an update.
|
||||
Actions → **Desktop Release** → version `0.0.0-test` (or any `X.Y.Z`) → publish **never**, or:
|
||||
|
||||
```bash
|
||||
gh workflow run release-local.yml --ref dev -f version=0.0.0-test -f publish=never
|
||||
```
|
||||
|
||||
Not a tag, not an update: the installers stay on the run as artifacts, and `legacy-update-bridge` does not run. It is not unsigned. macOS is signed and notarized, because the Apple secrets reach every run in this repository. Windows is signed through signtool with that same macOS certificate (`CSC_LINK` reaches every runner), not through Azure Trusted Signing, which only a tag uses. Linux signs nothing. A fork has none of these secrets, so its run signs nothing.
|
||||
|
||||
## Do not
|
||||
|
||||
- Tag `v2.0.1` before `VERSION` is `2.0.1` and the bump script has been committed.
|
||||
- Tag `v2.3.0` before `VERSION` is `2.3.0` and the bump script has been committed.
|
||||
- Tag `dev`, or any commit but the release pull request's merge on `main`.
|
||||
- Push a `v0.*` tag for this app (that line is the legacy desktop).
|
||||
- Publish the draft before Mac, Windows and Linux have uploaded, and before `legacy-update-bridge` is green.
|
||||
- Delete `legacy-update-bridge` while 0.0.x clients are still checking in. Watch `SurfSense-Setup-0.0.40.exe.blockmap` on the v0.0.40 release: while it climbs, legacy installs are still migrating.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue