Commit graph

231 commits

Author SHA1 Message Date
Anish Sarkar
c1a026e945 Merge upstream dev into feat/plugins-homepage
Resolves dev's sign-in failure codes (#2267) and its OCR ADR (#2295)
against this branch:
- dev's ADR 0062 keeps its number; this branch's become 0063, 0064 and
  0065, with every citation updated.
- A failed sign-in says why on the plugin's details, where this branch
  follows it, in dev's words for its code.
- dev's refusal tests keep the plugin waiting to sign in, as this branch
  saves it first, and its failure tests read the plugin's details.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 00:31:35 +05:30
Anish Sarkar
e9b495c0ee docs(plugins): a plugin's connection is a transport and one sign-in
ADR 0062 records the connection model: Streamable HTTP or SSE, found from
the server; no sign-in, headers, or OAuth registered in the spec's order or
as the user's own app; one Add screen with Advanced settings. The plugins
doc describes the built behaviour, and the connections proposal it replaces
is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 16:30:55 +05:30
Anish Sarkar
d928ec672e docs(plugins): propose connecting any remote MCP server with any sign-in
A connection is a transport, Streamable HTTP or legacy SSE, and one sign-in:
none, headers for any key, or OAuth that registers itself or uses the user's
own app. Detected from the server where it can be, shared by the Add custom
plugin dialog and the registry bot, with no migration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 03:50:12 +05:30
Anish Sarkar
d7850f4013 docs(plugins): local MCP servers wait for bundles, not custom plugins
Plugins stay remote MCP servers. A server started by a command is not added
as a custom plugin; local ones come with bundles, whose design is redone
first. The reasons move from the open questions into a decided section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 02:36:31 +05:30
Rohan Verma
efb156d814
Merge pull request #2265 from Cedric921/ci/release-dry-run-signing
ci(local): keep the macOS certificate off Windows and check Apple's credentials before every build
2026-10-17 18:11:20 -07:00
Cedric Karungu
1937f77949 docs(proposals): give the theme provider a resolved theme for Mermaid 2026-10-17 21:50:09 +02:00
Cedric Karungu
788549f43a docs(proposals): render Mermaid diagrams in the desktop app 2026-10-17 21:13:55 +02:00
Anish Sarkar
6bdbb969d3 docs(local): the Claude plan is architecture now, not a proposal
The work the Claude plan proposal laid out is built, so what is true now
moves to docs/architecture/claude-plan.md and the proposal is deleted, as
the docs process asks of the change that completes the work. What the
proposal promised and the code does not do yet is listed as Known gaps.
ADR 0057 says where the code stands, and the docs and code comments that
cited the proposal cite the architecture doc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 04:41:26 +05:30
Anish Sarkar
5cbbb06c4d fix(local): a Claude plan turn reads a large tool result Claude Code saved for it
Claude Code saves a tool result too large to give whole under the
session's tool-results folder and tells the model to read it there,
which the gate refused as outside the thread. The gate now lets the model
read that session's saved results alone, as opencode's rules leave its
tool-output folder open; another session's, Claude Code's settings and
any write stay refused. The comment and docs that said a raised
MAX_MCP_OUTPUT_TOKENS stops results being saved are corrected, and the
proposal notes that the tests running the real Claude Code will go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 03:39:16 +05:30
Anish Sarkar
8baa1865d5 fix(local): a Claude plan turn names a plugin unavailable only once Claude Code has tried it
The Claude turn asked Claude Code for its plugins' status right after it
started, when every one is still pending, and named them all unreachable.
It now reads them from the turn's init message, which Claude Code holds
until each has connected or failed. Both engines follow one rule: a
plugin is unavailable only from its engine's settled answer, within the
same 25 seconds, now Claude Code's MCP_TIMEOUT too, and for the same
three reasons, which move out of opencode's own module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 02:46:52 +05:30
Anish Sarkar
fd171c873f Merge remote-tracking branch 'upstream/dev' into feat/claude-plan 2026-10-17 02:22:00 +05:30
Anish Sarkar
9d2ea9c59a fix(local): the Claude plan row shows at once, and the dialog says to come back
The row waited on an endpoint that runs the user's claude twice. It is
split: GET /llm/connections/claude says what a plan fills and the hosts
still off without running Claude Code, and GET .../claude/status reads
Claude Code's state live, for the dialog alone. With Claude Code not
installed, the dialog now ends its steps with coming back and checking
again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:29:30 +05:30
Anish Sarkar
cfc2a3a6c2 feat(local): Agentic threads run on Claude Code for a Claude plan model
A Claude plan model now opens Agentic threads, with no opencode needed,
and its turns run on the user's own Claude Code with what opencode's do:
SurfSense's tools over the same endpoint, the thread's plugins, skills,
the folder rules, approvals, Stop, and the same frames on screen. A
running Claude Code is kept per thread and relaunched when a turn's
servers, rules or model change; an interrupted turn is read to its end
so it never answers the next message.

Changing models across engines hands the thread over: the engine taking
it resumes its own session, or opens one in the thread's folder, and is
given a note of the turns it missed; the thread reads back as one.

The tool, permission and step routes are mounted without opencode too,
behind the launch key, and a Claude thread is offered the tools that
return images.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 23:58:41 +05:30
Anish Sarkar
33480591d9 docs(local): the Claude agent does everything opencode's does
List each agent capability against both engines, and the two pieces of
opencode-only code that change so both get them. A Claude plan keeps
serving text generation only; what opens Agentic to it is the gate in
engine_choice.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 22:35:20 +05:30
Anish Sarkar
d2d23acd88 refactor(local): reach the agent through an engine interface
The chat, permission, step and workspace routes called opencode's thread
functions directly, so a second engine would have meant a branch at
each. They now ask the thread's engine through one protocol,
AgentEngine, and opencode's existing code is its only implementation,
with no change in behaviour. Recording a new thread's opencode session
moves from the chat router into the opencode engine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 22:01:37 +05:30
Anish Sarkar
111e0aa557 docs(local): settle the Claude agent's plugins, skills and sessions from spikes
Spikes against Claude Code 2.1.301 answered the agent's open questions.
Claude Code connects to plugins itself, as opencode does, so agent runs
get a proxy that also passes ready plugin hosts (ADR 0059, superseding
ADR 0057's network line); chat runs still reach only Anthropic. MCP
servers go in a file only the user can read, since the SDK otherwise
puts their headers on the command line. Denied tools go in
disallowed_tools, skills come through the plugins option, and the
result size limit is raised so nothing is written into ~/.claude. The
proposal now says why transcripts are mirrored: Claude Code deletes its
own after 30 days.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:45:22 +05:30
Cedric Karungu
763413973c ci(release): fail the notarization gate on a publishing run without secrets 2026-10-16 17:48:45 +02:00
Cedric Karungu
6cc61b1223 Merge branch 'dev' of github.com:MODSetter/SurfSense into ci/release-dry-run-signing
# Conflicts:
#	docs/architecture/packaging.md
2026-10-16 17:48:13 +02:00
Anish Sarkar
6c7ce81d72 docs(local): ask how the Claude agent reaches a plugin
The connection doc keeps Claude Code's proxy to Anthropic's two hosts,
while the agent doc gave Claude Code each plugin server, which needs
plugin hosts on that proxy. Record it as an open question for the agent
spikes, preferring plugins through SurfSense's tool endpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:00:34 +05:30
Anish Sarkar
e95b8d3ed6 feat(local): say so when a host is turned off during a reply
A Claude Code run stopped at once when a host it needs was turned off
mid-reply, but the chat called it unknown: chat had an error kind for
an egress refusal only before a reply. It now sorts EgressDeniedError
into egress_disabled, which says the host was turned off in Settings >
Network and to allow it there, in every language.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:58:24 +05:30
Anish Sarkar
c4763ae362 feat(local): keep every Claude Code process inside egress, and watch its hosts
The status check now runs through the Claude proxy with the environment
every run gets: it contacts nothing today, but a later Claude Code that
checks its sign-in online is held to consent too. The proxy logs a
refused host as Claude Code's, not opencode's, so a host a new release
tries on its own shows at once, and a claude_code test fails when the
installed binary refreshes its sign-in on a host SurfSense does not
allow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:53:13 +05:30
Anish Sarkar
aba52eaf79 feat(local): set up a Claude plan from Settings and onboarding
A "Use your Claude plan" row beside the ChatGPT one opens a dialog that
reads Claude Code's state live and says what to do: install it, run
claude and sign in, or update it, with Check again; once ready, Add
Claude models asks for both of its hosts in one egress question and
makes the connection. The egress prompt now asks about every host a
refusal or askEgress names at once, wording a Claude plan's two by what
each gets, which also makes ChatGPT's sign-in one question. Editing a
Claude plan only renames it, chat names its failures in Claude's terms,
and Electron opens Claude Code's install guide and no other page of
that site. Every new string is in all ten languages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:57:32 +05:30
Anish Sarkar
45cf8c546b build(local): install the Agent SDK without its own Claude Code
The SDK's wheels carry a 241 MB Claude Code; its source package carries
none. no-binary-package makes uv build the SDK from source, so no venv,
CI cache or frozen app holds that copy, and pnpm dev's predev does it
through uv run with nothing done by hand. A unit test, run on every
system in CI, fails if the installed SDK ever carries one again; it
replaces the spec-reading packaging test, which guarded a file that is
no longer installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:35:15 +05:30
Anish Sarkar
25c689db8e test(local): keep the Agent SDK's own Claude Code out of the app
The SDK's wheel carries a 241 MB Claude Code that PyInstaller leaves
out, since nothing asks for the SDK's data. A packaging test now reads
every spec and fails when any collector names the SDK, and the frozen
API test fails when a claude binary is in the bundle. Measured: the SDK
adds 0.22 MB to the frozen API and no Claude Code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:31:04 +05:30
Anish Sarkar
26fc208eb1 feat(local): send Claude Code's traffic through a proxy the API runs
Every Claude Code run goes through a loopback proxy the API serves,
which tunnels only api.anthropic.com, for model calls, and
platform.claude.com, where Claude Code refreshes its sign-in, each
while its egress is on. No run starts without the proxy. Connecting,
listing models, choosing one and generating all require both hosts,
named in one refusal; a host turned off mid-reply ends the run at
Claude Code's first retry rather than its tenth. The proxy reuses the
plugin proxy's tunnel with a host list of its own, so opencode never
reaches Anthropic and Claude Code never reaches a plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:21:05 +05:30
Anish Sarkar
edeb6b3701 feat(local): answer the app's chat on a Claude plan
A claude_plan connection now resolves to the Claude plan's generator,
over one pool of warm runs for the API process, so chat, thread titles,
the chat check and Studio answer through the user's own Claude Code.
The API closes every warm run when it stops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 18:02:54 +05:30
DESKTOP-RTLN3BA\$punk
6c2e2df61e perf(local): read born-digital PDFs from their text layer
Every PDF went through Docling's layout, OCR and TableFormer models, about
3 s a page (5.5 on table pages) plus an 11 s cold start and 1 GB of RSS.
A born-digital PDF already carries its text: pdfium reads it in about
7 ms a page with the page check.

A PDF is now read from its text layer when every page passes a per-page
check (no scans, scanner OCR layers, garbled or margin-only text, rotated
pages or filled form fields); otherwise Docling reads it as before. Pages
are joined with a blank line, soft hyphens are rejoined. Text-read PDFs
keep their figures through the figures-only pass: on first request, and at
idle priority when the file has images.

Measured through the upload route on an idle Ryzen 5800X: Executive
Summary (12 pp) 16.7 s -> 1.4 s, Operating Budget (46 pp) 125.6 s -> 9.0 s,
a 33-page spec 73 s -> 3.8 s. In a Basic-mode A/B on the Boston budget
corpus the shipped text scored 0.70 strict against 0.60 for Docling's
markdown; a second, smaller corpus is 2.9 points behind on table cells.

SURFSENSE_LOCAL_PDF_TEXT_LAYER=off restores Docling for every PDF.
ADR 0059.
2026-10-16 03:02:17 -07:00
Anish Sarkar
b4d92da44f feat(local): answer chat through the user's own Claude Code
ClaudePlanProvider is a Generator over Claude Code with every tool off:
SurfSense's system prompt in place of Claude Code's, text and thinking
streamed, a schema answered as structured output, the newest message's
images sent beside it, and plan, sign-in and run failures raised as the
errors chat already shows. A warm run per conversation, at most four,
answers the next message from what it holds; an edited conversation, a
busy run or another model starts from the whole conversation instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:37:49 +05:30
Anish Sarkar
eb6e94f7bb feat(local): connect a Claude plan from the user's own Claude Code
GET /llm/connections/claude reads Claude Code's state live: not
installed, too old, signed out, or ready with its account and plan.
POST makes the claude_plan connection once Claude Code is ready and
api.anthropic.com allowed, one per install. The connection serves chat
only, lists Claude Code's models by their full ids so the manifest's
Anthropic capabilities apply, and can only be renamed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 13:05:33 +05:30
Anish Sarkar
af5f15cc76 feat(local): let a connection be a Claude plan
Revision 0031 widens provider_connections.auth_kind to accept
claude_plan, the only schema change the Claude plan needs: it stores no
credential, and what Claude Code reports is read live. SQLite cannot
alter a CHECK, so the table is rebuilt from its own stored definition,
with every remote selection kept aside and put back, since dropping the
table cascades through selected_models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 12:32:43 +05:30
Anish Sarkar
abafc09ff7 feat(local): find the user's Claude Code and read its sign-in
locate_claude searches PATH, then where Claude Code's installers put it
on macOS, Linux and Windows; on Windows only a claude.exe counts, since
the Agent SDK refuses npm's claude.cmd. read_status runs --version and
auth status and reports not_installed, too_old, signed_out or ready,
treating a hang or unreadable answer as no answer. Tests run a stand-in
claude on every system.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 12:13:49 +05:30
Anish Sarkar
b7ce936b21 docs(proposals): structured output works on a Claude plan
Measured: Claude Code returns output_format answers through its own
StructuredOutput tool, valid against the schema in every run, and needs
more than one turn. A Claude plan model can therefore take the chat
router step and Studio's schemas; the open question is closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:28:33 +05:30
Anish Sarkar
a742536cf8 docs(adr): a Claude plan runs through the user's own Claude Code
ADR 0057 offers a Claude plan through the user's installed, unmodified
Claude Code, with no Claude credential held by SurfSense and no Claude
Code shipped. It supersedes ADR 0038's "no Claude subscription" and
narrows ADR 0050's plan sign-in to the ChatGPT plan. The proposal it
records is marked accepted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:20:37 +05:30
Anish Sarkar
3ad5dbba45 docs(proposals): resume each engine's own session across a model switch
A thread returning to an engine resumes that engine's session with a note
of only the turns since; a failed resume, a changed Claude account or a
stale session starts fresh with the full note. The note travels beside the
user's message, leaves out reasoning and superseded output, and is kept
with its turn. A history tool is rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:13:30 +05:30
Anish Sarkar
549e3ef5c2 docs(proposals): use your Claude plan through the user's own Claude Code
Chat and the agent reach a Claude plan by driving the user's installed
Claude Code through the Agent SDK. SurfSense holds no Claude credential
and ships no Claude Code. Chat runs it with no tools; the agent runs it
as a second engine behind SurfSense's tools and permission gate, with a
handover note when a thread switches engines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:00:45 +05:30
DESKTOP-RTLN3BA\$punk
be961a596c fix(local): stage audio.cpp again when it lacks a curated family
A checkout staged before canary_asr joined the pins kept its old server,
because staging returned early whenever any server was staged. Choosing
Canary 180M Flash then failed with 'unsupported model family hint:
canary_asr'. Staging now probes the staged server once per curated family
and compiles again when one is missing or the server does not start.
Without --strict a stale server is kept, with a warning naming what it
lacks, when no toolchain is installed, when the app holds the folder
(checked before compiling) or when staging again fails, so pnpm dev goes on.
2026-10-15 16:36:05 -07:00
Anish Sarkar
256c3eebb6 feat(local): add Canary 180M Flash as a second speech model
A light recogniser for English, German and French: 249.5 MB and a 783 MiB
peak on its first request (Apple M2), against Nemotron's 931 MB and 1,776 MiB.

- Build canary_asr into audio.cpp and curate it in the manifest, with
  cc-by-4.0 on the licence allowlist.
- A model that cannot detect the language is always told one: Canary told
  nothing translates into English. Its intent starts on English and auto is
  refused.
- The cleanup joins a comma or full stop written apart from its word, as
  Canary writes German.
- Spanish is not listed: audio.cpp v0.8.2 drops every "ñ" Canary writes.
- Settings' speech rows no longer show the memory while transcribing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 03:14:00 +05:30
DESKTOP-RTLN3BA\$punk
531c1f6810 fix(local): keep a ChatGPT plan out of speech settings, and docs that match v1
After dev's one-click ChatGPT sign-in, the local-only speech section must
still offer no remote model; a test now proves it with a plan that claims
the slot, and fails without the localOnly gate. The docs drop two stale
Known gaps (no speech screen; no transcription test), name the settings nav
as it reads, count migrations to 0030, mark steps 2 to 7 done in the
proposal, and note the curated recogniser's type in the catalog proposal.
2026-10-14 15:04:38 -07:00
DESKTOP-RTLN3BA\$punk
d9c0a86d98 Merge remote-tracking branch 'origin/dev' into feat/local-speech-recognition
dev took ADR 0055 for the plugin registry's Verified rule, so the speech
recognition record is now 0056, Accepted with the change it decides, and
every link to it follows. Its 'Where the code stands' section went: the
code now holds the whole decision.
2026-10-14 14:51:34 -07:00
Cedric Karungu
762da46c82 ci(local): keep the macOS certificate off Windows and check Apple's credentials before every build 2026-10-14 20:44:01 +02:00
DESKTOP-RTLN3BA\$punk
553048c02e feat(local): transcribe recordings as sources in the ingest worker
Step 5 of the local speech recognition proposal, backend half. A recording
uploaded as a source becomes a timestamped transcript in its content, so
chunking, search, citations and the agent take it unchanged, without one long
file holding every other upload behind it on the one-thread ingest queue.

- Upload: .wav .mp3 .flac .ogg .oga with server-owned audio/* types, checked
  by their bytes (RIFF/WAVE, ID3 or an MPEG frame, fLaC, Ogg with a Vorbis
  first packet). Ogg Opus is rejected per file: "Opus support comes in a
  later update". can_hold_figures is false for audio/*, so Docling never
  reads a recording.
- worker/audio_decode/: the worker's own binary, started as
  `worker --decode-audio <in> <out>` ahead of the consumer's imports, checks
  the disk from the file's header length, streams it through pyminiaudio
  (MIT) to 16 kHz mono s16 and exits when its parent does. A malformed file
  kills the child, not the consumer. miniaudio joins pyproject, uv.lock and
  worker.spec (with its licence metadata).
- worker/ingestion/audio/, reached from parsing before Docling: readiness
  through the API's status route, the decode, Silero over the file a minute at
  a time (the API's reader, now able to stream), spans padded and joined,
  windows of at most 28 s cut at the longest pause or a run's quietest
  100 ms, each window sent to /internal/transcription/windows with retries and
  waits, windows.jsonl and progress.json as the checkpoint, a progress event
  per window, then the header and [hh:mm:ss] paragraphs, segments.json and
  transcription {language, duration_s, model} in the row's metadata.
- Yielding: after a window or a 503 waiting, once a minute has passed and a
  source that is not a recording is pending, the job raises JobYieldedError;
  the pipeline sets the row back to pending (jobs.pause_job) and queues it at
  PRIORITY_BULK. Recordings never yield to each other.
- No speech model: the job fails without retry and marks
  transcription.awaiting; choosing a speech model, by the route or an install
  with select, queues every waiting recording again. Refusals (no room, no
  speech, no audio, undecodable, memory short for 10 minutes) are
  CancelExecution(retry=False); every other retry resumes from windows.jsonl
  when the model and language intent match.
- DocumentRead gains transcription {language, progress, awaiting} for audio
  rows, progress read from progress.json; InternalEvent and the SSE frame
  gain an optional transient progress {done_s, total_s}.

Measured here: an 8.6-minute file from pending to ready in 60 s at 5.83% word
error; an hour of MP3 decodes in 3.6 s; miniaudio's linear resampler read the
44.1 kHz copy at 5.91% against scipy's 5.83%, under the proposal's 2% relative,
so it stays. Docs: documents.md gains Recordings; data-model.md, overview.md,
selection.md and packaging.md follow; the proposal records the two step 5
measurements.
2026-10-14 08:31:45 -07:00
DESKTOP-RTLN3BA\$punk
6c952c226b fix(local): answer a hung speech server with 503 and keep numbers in CJK text
Review of the speech recognition API slice.

- A transcription request past its deadline raised httpx's ReadTimeout out of
  the client, so the phrase and windows routes answered a bare 500 instead of
  503 transcription_unavailable, which the worker retries and the composer
  explains. Any HTTP error the client meets, and a 200 with no JSON text, now
  become TranscriptionUnavailableError; a timed-out request is not asked again.
- The cleanup's character-level collapse for zh, ja, yue and th counted a
  number's digits as characters, so 100000 in a Chinese transcript became 10,
  and Hmmmm became Hm. A run of digits or Latin letters now counts as one unit.
- A WAV cut mid-sample passed the check with half a sample, which made Silero
  fail (the clip then skipped the speech check) and reached the server as is.
  The check keeps whole samples only.
- The keepalive and close routes were sync handlers, so they changed the
  dictation sessions from the threadpool while the gate read them on the event
  loop. They are async now, like every other use.

Docs: the proposal's Transport table gains the windows route's `busy` wait and
a phrase's 409 transcription_needs_memory after a hold, both of which the step
documented in admission.md only; the cleanup rule names the unit; admission.md
says a timed-out request answers 503.
2026-10-14 06:39:44 -07:00
DESKTOP-RTLN3BA\$punk
939de654fc feat(local): transcribe speech through one gate in the API
Step 4 of the local speech recognition proposal: the API slice that the
composer's mic and recordings as sources will call. The transcription server
runs one model and one request at a time, and only the API reaches it, so
the API decides the order: a speaker never waits behind a recording, and a
podcast or an image gets its memory back.

- modules/llm/transcription/: the dictation routes (status, open, phrases,
  keepalive, close) under /llm/transcription, and the ingest worker's
  POST /internal/transcription/windows, which answers 503 waiting with its
  reason (dictation, hold, memory with both sizes, or busy) after 30 s.
- The gate sends one request at a time, a phrase ahead of any window. An open
  dictation (until its last phrase, 20 s without one, or 6 minutes) starts no
  window. A text-runtime hold, which /internal/models/text/yield now takes,
  starts no window, waits for the request in flight, and unloads the speech
  model by id; dictation goes on behind the memory check.
- Memory: a loaded model needs no check; otherwise the entry's measured peak
  plus 1 GiB against the operating system's figure, refused in a sentence.
  memory.py's check now takes the verb ("Transcribing", "Voicing").
- providers/audiocpp/transcription.py behind a new Transcriber protocol:
  the WAV from memory as multipart to /details, a deadline of
  max(120 s, 20 x the audio), one retry after 2 s on a dropped connection or
  a 503, and the model held in use so a delete mid-request is refused.
- Phrases are checked with the standard library and by Silero VAD on the
  onnxruntime already in the API; under 250 ms of speech never reaches the
  model. Text loses Nemotron's locale tags, the last naming the language, and
  goes through the proposal's fail-open cleanup.
- The packaging test found French and Hindi sentences untagged, past the
  proposal's 1 in 10, so with no tag under auto the language is read from a
  script only one listed language writes.
- scripts/fetch_silero_vad.py stages Silero's 16 kHz ONNX build and its MIT
  licence from v6.2.3's commit, sha256-checked, into the models pack; build:vad
  and the release workflow run it beside the bundled voice.
- tests/packaging/test_audiocpp_transcription.py has Kokoro voice six
  languages and the real server write them down through the app's client
  (peak 1,743 to 1,774 MiB here, against the entry's 1,776); build-audiocpp.yml
  runs it and the config test, with the recogniser cached by its hash.

Docs: admission.md gains speech recognition and the hold, studio.md the hold's
effect, packaging.md the pack and the tests, ADR 0055 the gate.
2026-10-14 05:52:54 -07:00
DESKTOP-RTLN3BA\$punk
a08ef99ee4 feat(local): offer Nemotron 3.5 ASR for a new speech recognition slot
Step 3 of the local speech recognition proposal, the backend half. A user
can now download Nemotron 3.5 ASR Streaming 0.6B and choose it for
speech_to_text, and the API writes the transcription.json that step 2's
watcher starts the second audio.cpp server from. Nothing calls the server
yet, which selection allows.

- ModelType.SPEECH_TO_TEXT is a local-only slot: a model nothing recognises
  and every connection stop short of it, since no remote speech client
  exists, and a confirmed unlisted model cannot get round that.
- Revision 0030 rebuilds selected_models so audiocpp may hold audio_gen or
  speech_to_text and the type CHECK lists six slots. It carries every column;
  0018's rebuild predates 0019's settings, so copying it would drop the voices
  users added. Downgrading drops only a speech_to_text row.
- The manifest gains a reviewed transcription block (sample rate, measured
  peak, window, languages, the family's locales, detected-only locales,
  request options). The schema refuses a locale or option the pinned
  nemotron_asr does not take, since the server fails such a request, and makes
  each audio.cpp entry carry the block its type names.
- Nemotron's Q8_0 is pinned at audio-cpp/audio.cpp-gguf f861648, validated on
  v0.8.2, listed for 26 languages; OpenMDW-1.1 joins the licence allowlist
  with its "without restriction" clause quoted.
- transcription.json names each installed recogniser with
  default_request_options (keep_language_tags) and is written only when its
  bytes change, so an API start, a voice install or a delete never restarts
  that server. A recogniser's install leaves server.json alone, which a
  podcast may be voicing from; a delete the server holds open rewrites
  transcription.json unchanged, which is what restarts it.
- Rows carry a transcription summary, every engine leads with every name
  chosen for any of its slots, and an install with select fills the row's own
  slot rather than the engine's first.
- resolve_speech_to_text() returns the selection and its installed
  recogniser; GET and PUT /llm/selection/speech_to_text/language keep the
  language intent under the row's transcription settings, auto or a listed
  code, and answer 409 needs_speech_model without one.

Docs: selection.md, catalog.md, data-model.md and overview.md, and glossary
rows for speech recognition, dictation and transcript.
2026-10-14 04:13:40 -07:00
DESKTOP-RTLN3BA\$punk
75603e3b5e fix(local): never leave a second speech recognition server running
The transcription watcher stops the old server and then starts the new one,
and that stop can take seconds while the server exits. Two cases started a
server that nothing would ever stop:

- A poll that came during the stop saw the rewritten config, found no child
  under the name (the supervisor forgets it before waiting) and started one;
  the first poll then started another over it, orphaning the first with its
  port and its model's memory. Polls now skip while one is still running.
- A quit that began during the stop stopped only the children it saw, and
  the poll then started a fresh server after it. A start now checks for the
  quit again.

The proposal's code list also names shared/config.py, which step 2 changed.
2026-10-14 02:17:26 -07:00
DESKTOP-RTLN3BA\$punk
5d4dc25312 feat(local): stage speech recognition's audio.cpp server, not yet started
Step 2 of the local speech recognition proposal. Nemotron 3.5 ASR runs on
the audiocpp_server the app already ships for voices, as a second process
with its own config, port and crash restart, so a voice install, a
podcast's unload or a busy voice model never touches a transcription.

- pins.mjs builds nemotron_asr beside the voice families, so its spec is
  staged too; staging now starts the server once per curated family on an
  empty model package and fails unless that family's loader runs, which
  --list-devices could not show. It also prints how long staging took.
- build-audiocpp.yml gains a macOS job that stages upstream's archive the
  same way; the Linux and Windows gates re-run because the pin changes the
  cache key.
- Electron has the audiocpp-transcription sidecar (transcription.json, its
  own free port, the voices' flags without eSpeak) and a watcher that starts,
  restarts and stops it with that file, and restarts it after a crash at
  most once per 10 s, giving up after 3 crashes in 5 minutes until the file
  changes.
- The API alone receives SURFSENSE_LOCAL_TRANSCRIPTION_BASE_URL, read into
  LLMSettings.transcription_base_url; workers will go through the API.

Nothing writes transcription.json yet, so the server never starts.
ADR 0055 records the decision; packaging.md and overview.md describe it.
2026-10-14 01:54:13 -07:00
Anish Sarkar
e98038ce0b
Merge pull request #2255 from AnishSarkar22/feat/remote-plugins
docs(plugins): remote plugins move from proposal to architecture, with a guide to listing one
2026-10-14 14:03:35 +05:30
Anish Sarkar
b84da18757 docs(plugins): fold the built remote plugins into architecture, keep only what is not built as the proposal
The core/ and remote/ proposals described work that has shipped. What they
said that the architecture docs lacked now lives there: the MCP client's
rules, what the gateway does today, what a plugin can reach, adding a plugin
to the host, who changes proprietary/, and the backend's copy of the
scrapers. proposals/plugins/README.md now holds only what is not built: the
chat router step, jobs and scaling on the host, organisation policy, later
MCP features and the open questions. Bundles stay deferred, with their
registry entry moved in. ADRs 0051, 0053 and 0054 are accepted, and links
into the deleted files point at the architecture docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:47:47 +05:30
DESKTOP-RTLN3BA\$punk
55b3ddc2e9 docs(proposals): local speech recognition on a second audio.cpp server
Recordings become sources and the composer gets a mic, both on the user's
CPU through the audio.cpp server the app already ships for podcast voices.
Nemotron 3.5 ASR Streaming 0.6B is the model: measured on audio.cpp v0.8.2
beside Qwen3-ASR-0.6B, it held half the memory and ran three times as fast.
2026-10-14 01:11:21 -07:00
Rohan Verma
a4a5d9aa5e
Merge pull request #2147 from vintcessun/docs/1997-cjk-keyword-proposal
Some checks are pending
Plugins Registry / Check registry (push) Blocked by required conditions
Plugins Registry / Detect changes (push) Waiting to run
Plugins Registry / Refuse hand edits to the signed list (push) Blocked by required conditions
Plugins Registry / Sign the registry (push) Waiting to run
docs(local): propose keyword search for Japanese and Chinese
2026-10-13 15:13:37 -07:00