SurfSense/docs/architecture/overview.md

Desktop app overview

SurfSense is a desktop app for research over your own documents: add files, ask questions that are answered with citations into them, and turn them into deliverables such as summaries, slides, quizzes and podcasts. It serves one person on one machine and works offline: the parser, the embedding model and the model runtimes run locally, and everything the app knows lives on this machine, in one SQLite file and the files under its data directory. This page is the map of surfsense_local/; each feature has its own page, listed at the end.

Code: surfsense_local/backend/, surfsense_local/frontend/, surfsense_local/electron/ Decisions: ADR 0004, ADR 0005, ADR 0008, ADR 0009, ADR 0011, ADR 0016, ADR 0017

Bird's-eye view

A file is uploaded, parsed to markdown, cut into passages and embedded. A chat turn retrieves the passages nearest the question, and a model answers from them, citing each one. A Studio job hands the selected documents' text to a model and stores the result as a document, with a file for ten of its twelve formats. Every design choice follows from who this is for:

• One user. There are no accounts, no auth and no users table. The API binds to 127.0.0.1 and accepts any origin, because the packaged renderer loads from file://.
• Offline. Docling parses, bge-small embeds on onnxruntime, llama-server generates text, sd-server generates images and audio.cpp's server voices podcasts, all on this machine. Remote destinations (Hugging Face for model search and downloads, and each remote host a connection points at) stay off until the user allows them (egress.md). The app carries no telemetry or analytics code.
• SQLite. surfsense.db holds every table, the search index included: FTS5 for keywords, sqlite-vec for vectors. huey.db holds the job queues. Next to the hosted stack there is no Postgres or Zero, no Celery or Redis, no LangGraph and no Docker.

Processes

Electron main ─┬─ api            FastAPI on 127.0.0.1, free port    ─┐
               ├─ worker-ingest  Huey consumer, "ingest", 1 thread   ├─ surfsense.db, huey.db
               ├─ worker-studio  Huey consumer, "studio", 4 threads ─┘
               ├─ llamacpp       llama-server in router mode
               ├─ sdcpp          sd-server, started once an image model is chosen
               ├─ audiocpp       audiocpp_server, started once the API names an audio model
               └─ opencode       opencode serve, started once the API writes its configuration

BrowserWindow (Vite SPA) ── HTTP ──> api
api ── admission ── HTTP ──> llama-server, remote OpenAI-compatible endpoints
worker-studio, opencode ── POST /internal/models/text/generate, /agent/model/v1 ──> api
worker-studio ── HTTP ──> sd-server, audiocpp, remote image and speech endpoints
worker-ingest, worker-studio ── POST /internal/events ──> api ── SSE /workspaces/{id}/events

• Electron's main process starts four sidecars at boot and supervises them (index.ts, sidecars/): the API, one Huey worker per queue, and llama-server. Packaged builds run frozen binaries from the app's resources; pnpm dev runs main.py and worker.py <queue> on the backend's .venv interpreter, synced by predev's own uv run scripts. Not through uv run: Windows kills Electron's children with it but not theirs, so after a Ctrl-C the last session's Studio worker, still reading the queue, took the next jobs.
• llama-server starts whenever its pinned build is staged, in dev too (pnpm build:llamacpp, which predev runs). It reads per-model arguments from a preset file once at startup, so Electron restarts it when the API rewrites that file (local-models/runtime.md).
• sd-server takes its model as startup arguments, so it cannot start at boot. Whenever its pinned build is staged, in dev too (pnpm build:sdcpp, which predev runs), watchImageModel asks the API every 5 seconds which files to run, and starts, restarts or stops it on a change. The API names a model only while a Studio job needs one and for up to 5 minutes after, so its weights are not held beside the chat model all session (studio.md).
• audiocpp_server refuses an empty model list, so it starts only once the API has written audio/server.json under the data directory, in dev too (pnpm build:audiocpp, which predev runs). On Windows and Linux that script compiles audio.cpp, and without a C++ toolchain the app runs without local audio (packaging). Electron checks that file every 5 seconds and restarts the server when it changes. Electron sets the machine-wide flags: the CPU backend, half the logical cores up to 8, one loaded model, and an unload after 5 idle minutes. The API writes that file whenever an audio model is installed or deleted, and at startup (local-models/catalog.md). The Studio worker voices podcasts there, at SURFSENSE_LOCAL_AUDIO_BASE_URL, and unloads the model when a podcast ends (studio.md).
• Only the API gates the window. Electron waits up to 60 seconds for /health and gives up at once if the API exits. llama-server is best-effort; its state shows through /llm/providers.
• opencode, the agent's engine, starts only once the API has written agent/opencode.json under the data directory, in dev too (pnpm build:opencode, which predev runs, stages it only while opencode is on: agent). Electron checks for that file every 2 seconds: it starts opencode once the file is there, stops it when the file goes, and restarts it after a crash, at most once every 10 seconds. A rewrite does not restart it, because opencode reads the file per folder the first time the folder is used; the API makes it read the file again through POST /global/dispose (agent proposal). Electron removes the file at boot, so each run's API writes its own. At boot Electron picks opencode's port and a random password, which it passes to the API alone as SURFSENSE_LOCAL_OPENCODE_URL and SURFSENSE_LOCAL_OPENCODE_PASSWORD. opencode's environment is built from a few system variables rather than inherited, with its home and XDG folders under agent/opencode/, its own fetches switched off and its proxy pointed at the discard port (sidecars/opencode.ts). serve keeps running when its parent is killed, so Electron records the pid, port and password in agent/opencode-process.json, and the next boot stops a recorded process that still accepts that password.
• Before a quit, whether from the menu or by closing the last window, Electron asks the API how many chat replies are being written; with any, it asks the person, and on Quit has the API store each reply before anything is stopped (chat). Cancel keeps the app open.
• On macOS and Linux each child runs in its own process group. On quit Electron sends SIGTERM and, after 5 seconds, SIGKILL; the group is also killed as soon as the sidecar itself exits, because a child that ignores SIGTERM would outlive the app. On Windows it kills the process tree. A single-instance lock hands a second launch to the first window, because two sets of sidecars would fight over the SQLite file.
• The Python sidecars are configured through SURFSENSE_LOCAL_* variables: the API's host and port, the data and models directories, the llama-server and sd-server addresses, the images folder, audio.cpp's address and folder where it is staged, and SURFSENSE_LOCAL_SECRET, the key that encrypts stored API keys (connections.md). No other sidecar receives the secret. The API alone also gets SURFSENSE_LOCAL_SHELL_PID, Electron's own pid, the root of what Settings › Resources counts as the app (resource-usage.md).

Layer boundary

Concern	Runs in
HTTP routes, OpenAPI, the chat and events streams	api
Migrations and the default workspace, at startup	api
Upload streaming, dedup and enqueueing	api
Query embedding and retrieve() for chat	api, in process
Chat runs, the chat stream and thread titles	api
Admission to the local runtime, and text generation for every other process	api (admission)
Model catalog, downloads and the hardware probe	api
Parse, chunk, embed and index a document	worker-ingest
Generate and render an artifact, index its body	worker-studio
Sidecar lifecycle, the keychain secret, updates, opening files natively	Electron main

The rules that keep the processes out of each other's way:

• The API owns the schema. upgrade_to_head() runs in the API's lifespan before anything else, and the workers only read and write rows (ADR 0005). The API then seeds a workspace named "My Workspace" if none exists.
• A job is enqueued after the commit that writes its row. The worker is another process and would otherwise look for a row the request had not committed.
• Workers change status through worker/jobs.py: begin_job marks a row processing unless it was cancelled, raise_if_cancelled runs between steps, and finish_job writes the terminal status only if a cancel has not won the race. A running job is never killed; it notices a cancel at its next step.
• Every connection runs in WAL mode with a 5-second busy timeout, foreign keys on and sqlite-vec loaded, and opens transactions with BEGIN IMMEDIATE, so a read-then-write waits for the lock instead of failing. The price is that no transaction may stay open across a slow call. The API raises if a request opens one on the event loop, and async handlers run each stretch of session work through transact() (api/dependencies.py).
• Ingest runs one job at a time because it saturates a CPU and writes to the file the API serves from. Studio mostly waits on a model, so it runs four. Separate queues keep an import from sitting in front of a summary (ADR 0008).

Freshness

• Workers call POST /internal/events after each status change (worker/notify.py): ingest sends a documents event keyed by document id, Studio an artifacts event keyed by artifact id. The notice is best-effort with a 2-second timeout; losing one costs a live update, never the job.
• The API announces each reply, chat or agent, starting, changing state and ending on the same stream, as a chat-runs event keyed by thread id with status running, queued, needs-approval or done (chat).
• The API notifies of its own document changes on the same stream, straight to the broker: a note written, an upload, a rename or edit, a retry, a cancel, and a delete, whose status is deleted (api/notify.py). So a change reaches an open window whoever made it: the window itself, a plugin, or a second window.
• The API fans each notice out on GET /workspaces/{id}/events (modules/events/) as a named SSE event whose data is {"ids": [...], "status": "..."}. The stream opens with a : connected comment and sends : ping after 15 idle seconds. The broker is an in-memory map, which holds because one uvicorn process serves the app.
• The frontend holds one subscription per workspace, shared by the lists that listen (features/workspaces/workspace-changes.ts), and closes it with the last of them. The sources panel reloads its list on each documents event and the Studio artifact list on each artifacts event; both reload each time a dropped stream is back, for what changed while nothing listened. Because a notice can be lost, each list also refetches every 10 seconds while any of its rows is pending or processing, and stops when none is: a lost notice leaves a row stale for at most that long, and nothing is requested on a timer while nothing is in flight.

Data directory

~/.surfsense/                 ~/.surfsense-dev under `pnpm dev`
├── surfsense.db              every table and the search index
├── huey.db                   the ingest and studio queues
├── models/                   GGUF weights and llama-server's models.ini
├── images/                   sd-server weights (hosts with sd-server staged)
├── electron/                 Electron's userData: secret.bin, updates.json, window and theme prefs
├── agent/                    opencode.json, which the API writes, and opencode's own home (hosts with opencode staged)
└── data/workspaces/<id>/
    ├── documents/<id>/       the upload, under its own name
    ├── chats/<id>/           images a thread's turns carried
    ├── artifacts/<id>/       an artifact's rendered file, named by role
    └── agent/                each agent thread's folder and the text its sources link to

• The bundled embedding, parser and voice models are read from SURFSENSE_LOCAL_MODELS_DIR: the app's resources when packaged, backend/models under pnpm dev, and <data dir>/models for a bare uv run.
• Paths under data/ are built from row ids. The only part of a user's filename that reaches the disk is a validated extension.
• A bare uv run with no SURFSENSE_LOCAL_SECRET writes its key to secret beside the database.
• SURFSENSE_LOCAL_DATA_DIR, SURFSENSE_LOCAL_HOST and SURFSENSE_LOCAL_PORT override the defaults. Tables and files are detailed in data-model.md.

Frontend

• A Vite and React SPA with Tailwind and shadcn/ui components in components/ui/, built on Base UI in shadcn's base-nova style; assistant-ui drives the conversation, and still depends on Radix. Packaged, Electron loads frontend/dist/index.html from disk, which is why Vite builds with relative asset paths. In dev it loads the Vite server on port 5173.
• The API's address comes from the preload: Electron passes it as a command-line argument and preload/index.ts exposes it as window.surfsense.apiUrl. In a bare browser there is no preload, requests stay root-relative, and the Vite dev server forwards /health, /llm, /workspaces, /chat and /artifacts to 127.0.0.1:8000.
• The preload bridge is the renderer's only other channel: opening or revealing an original file natively, opening an external link, the platform name, updates, theme, the title bar, this session's log and unexpected sidecar-exit notices. The renderer never sees Node or the sidecars directly. A source with a registered viewer (PDF first) is fetched from the API and previewed in place of the left sidebar; other file types keep the native open action (documents.md).
• API calls go through request() in lib/api.ts, except the Studio viewers, which fetch an artifact's file bytes directly; an <img>, <audio> or download link takes its absolute address from apiUrl(). request() prefixes the address, turns an error body into an ApiError with its code, and when a user action is refused with 403 egress_disabled it asks for consent and retries once.
• TanStack Query holds most server state: threads and messages, the citation panel, the model catalog and settings, an open artifact. The sources list and the Studio artifact list are component state, reloaded on the events above.
• There is no router. app/app-bootstrap.tsx asks GET /llm/onboarding, then renders either onboarding (local-models/selection.md) or the lazily loaded dashboard: the workspace rail, threads and sources on the left, the conversation in the middle, and Studio, artifacts and the citation panel on the right.
• Code is grouped by feature under src/features/. Each HTTP contract lives in one feature's api.ts, next to its hooks and components, and other features import it from there.

Codemap

• backend/api/: create_app() and its lifespan (migrations, the default workspace, a background catalog warm-up), the per-request session and transact().
• backend/modules/: one folder per feature, holding whichever of its models, schemas, router and service it needs: workspaces, documents, chunks, chat, artifacts (Studio), events, llm (runtimes, catalog, connections, selection), egress, agent, license, migration (import) and health. shared.db.import_models() imports every model at startup, because relationships name their targets as strings.
• backend/worker/: consumer.py drains one queue per process, jobs.py holds status transitions and cancellation, notify.py the change notice, and ingestion/ and studio/ the two pipelines.
• backend/shared/: configuration, the engine and its pragmas, upgrade_to_head(), the two Huey queues, retrieve(), and the secret that encrypts stored keys.
• backend/alembic/: revisions 0001 to 0012, all hand-written. env.py has no target_metadata, so autogenerate cannot run by accident.
• frontend/src/features/: chat, agent, sources, studio, workspaces, dashboard, models, onboarding, settings, egress, license, migration, updates and feedback.
• electron/src/main/: index.ts (boot, window, IPC and the image-model, preset, audio and agent watchers), sidecars/ (the supervisor and one spec per sidecar), session-log/ (this run's output, for issue reports), secret.ts, updater.ts and document-files.ts.

Where to read next

• Data model: tables, the search index, revisions, files on disk.
• Documents: workspaces, upload, notes, the ingest pipeline.
• Search: retrieve().
• Embedding model: the model that embeds the library, chosen once at onboarding.
• Chat: grounding, the stream, runs, citations.
• Agent: a thread opencode answers in steps, and how it is kept to loopback.
• Model capabilities: the measured level each chat model gets, and what follows from it.
• Studio: artifact formats, jobs, viewers.
• Connections: OpenAI-compatible endpoints and where keys live.
• ChatGPT subscription: a connection signed in with a ChatGPT account, answered through the Responses API.
• Local models: runtime, fit, admission and parallel slots, catalog, selection and onboarding.
• Localization: the interface in English, Japanese and German.
• Egress, import, license in the app, license portal, updates, about, issue reports, resource usage, packaging, sunset.
• Contracts between the trees.

Known gaps

• The sources and Studio lists reload themselves on a workspace event rather than through TanStack Query: they are component state, so nothing else that shows a document or an artifact is invalidated by the same event.