fix(local): stage audio.cpp again when it lacks a curated family

A checkout staged before canary_asr joined the pins kept its old server,
because staging returned early whenever any server was staged. Choosing
Canary 180M Flash then failed with 'unsupported model family hint:
canary_asr'. Staging now probes the staged server once per curated family
and compiles again when one is missing or the server does not start.
Without --strict a stale server is kept, with a warning naming what it
lacks, when no toolchain is installed, when the app holds the folder
(checked before compiling) or when staging again fails, so pnpm dev goes on.

docs/architecture/packaging.md

@@ -73,7 +73,7 @@ The release workflow runs the four scripts directly. Without the parser pack, Do
   - Windows downloads upstream's Vulkan archive, checked against its pinned SHA-256, and copies the MSVC and OpenMP runtimes beside it, which the archive needs and does not carry.
   - Linux and macOS compile the pinned commit, because upstream's Linux archives need glibc 2.38 and its macOS archive needs macOS 26.0. The recipe, `scripts/sdcpp/recipe.mjs`, builds Vulkan on Linux, with one ggml library per micro-architecture, and Metal on macOS, for 13.3.
   - Compiling needs CMake, a C++ compiler and the Vulkan SDK (`glslc` and headers) on Linux, or Xcode's command line tools on macOS; Windows needs Visual Studio 2022 or newer with the C++ tools, for the runtime it ships. Without them the script stages an empty folder and says why, and the app runs without local images. Release CI and `pnpm dist` pass `--strict`, which fails instead, naming everything missing and the command that installs it with the machine's package manager.
-- `scripts/audiocpp/stage.mjs`, which `build:audiocpp` runs, stages audio.cpp's `audiocpp_server` into `electron/audiocpp/` with the model specs of the five curated families, `FAMILIES` in `scripts/audiocpp/pins.mjs`: Kokoro, Supertonic and Kitten for voices, and Nemotron's `nemotron_asr` and Canary's `canary_asr` for speech recognition, which the same binary serves from a second process ([ADR 0056](../adr/0056-speech-recognition-runs-on-a-second-audiocpp-server.md)). It adds eSpeak-ng 1.52.0 from the pinned `espeakng-loader` wheel, which Kokoro and Kitten phonemise through (Kokoro finds it through the server's environment, Kitten through `server.json`, [`local-models/catalog.md`](local-models/catalog.md)), and eSpeak-ng's GPL licence text, which the wheel does not carry. Before it swaps the folder into place, it runs `--list-devices` from it, then starts the server once per curated family on an empty model package (`scripts/audiocpp/built-families.mjs`): a family the build has fails inside its own loader, and one it lacks is refused by name first, so a family missing from the build stops the stage. It prints the staged size and how long staging took. A server staged before a family was added lacks it, and `stage.mjs` keeps a staged server as it is, so delete `electron/audiocpp/` once to pick the family up.
+- `scripts/audiocpp/stage.mjs`, which `build:audiocpp` runs, stages audio.cpp's `audiocpp_server` into `electron/audiocpp/` with the model specs of the five curated families, `FAMILIES` in `scripts/audiocpp/pins.mjs`: Kokoro, Supertonic and Kitten for voices, and Nemotron's `nemotron_asr` and Canary's `canary_asr` for speech recognition, which the same binary serves from a second process ([ADR 0056](../adr/0056-speech-recognition-runs-on-a-second-audiocpp-server.md)). It adds eSpeak-ng 1.52.0 from the pinned `espeakng-loader` wheel, which Kokoro and Kitten phonemise through (Kokoro finds it through the server's environment, Kitten through `server.json`, [`local-models/catalog.md`](local-models/catalog.md)), and eSpeak-ng's GPL licence text, which the wheel does not carry. Before it swaps the folder into place, it runs `--list-devices` from it, then starts the server once per curated family on an empty model package (`scripts/audiocpp/built-families.mjs`): a family the build has fails inside its own loader, and one it lacks is refused by name first, so a family missing from the build stops the stage. It prints the staged size and how long staging took. A server already staged is kept only when the same check finds every curated family in it (`scripts/audiocpp/staged-server.mjs`); one staged before a family joined the pins is compiled and swapped again, since it would answer that family's model with "unsupported model family hint", and so is one that does not start at all. Without `--strict` the stale server stays, with a warning that names what it lacks, when no toolchain is installed, when the app holds `electron/audiocpp/` (checked before compiling), or when staging again fails, so `pnpm dev` goes on with every family it has.
   - macOS downloads upstream's archive, checked against its pinned SHA-256.
   - Windows and Linux compile the pinned commit, because upstream's Linux archives need glibc 2.38 and its Windows archive compiles AVX-512 into the executable. The recipe, `scripts/audiocpp/recipe.mjs`, builds only the CPU backend, since the server runs with `--backend cpu`, with one ggml library per micro-architecture and only the curated model families.
   - Compiling needs CMake and GCC 13 or newer on Linux, or Visual Studio 2022 or newer with the C++ tools on Windows. On Windows the build names Visual Studio's generator (`msvc-runtime.mjs`), since a CMake from MinGW or MSYS2 defaults to Ninja, which refuses `-A x64` and finds no compiler outside a Visual Studio prompt. Without them the script stages an empty folder and says why, and the app runs without local audio. Release CI and `pnpm dist` pass `--strict`, which fails instead, naming everything missing and the command that installs it with the machine's package manager.

docs/proposals/local-speech-to-text.md

@@ -373,7 +373,7 @@ The runtime licence policy keeps GPL and LGPL code out of the app's own binaries
 
 ## Packaging and CI
 
-- [`pins.mjs`](../../surfsense_local/electron/scripts/audiocpp/pins.mjs): `FAMILIES` gains `nemotron_asr`; the recipe compiles only `FAMILIES` and stages their specs. `stage.mjs` returns early when a server is staged, so developers delete `electron/audiocpp/` once.
+- [`pins.mjs`](../../surfsense_local/electron/scripts/audiocpp/pins.mjs): `FAMILIES` gains `nemotron_asr`; the recipe compiles only `FAMILIES` and stages their specs. `stage.mjs` keeps a staged server only while it has every family in `FAMILIES`, and stages again otherwise.
 - [`build-audiocpp.yml`](../../.github/workflows/build-audiocpp.yml) re-runs the Linux `objdump` gate and the Windows per-CPU check with the new family, checks that upstream's macOS v0.8.2 archive contains it, and records the staged size and compile time.
 - `tests/packaging/test_audiocpp_transcription.py`, added to the workflow's path filter: Kokoro voices known sentences in en, es, fr, hi, pt-BR and zh, the transcription server writes them down through the app's client, and the test asserts error and detected-language thresholds and records peak memory. No fixture with a licence of its own is needed. The model file is cached by its manifest sha256, unlike the voicing step's downloads, which run every time (step 4).
 - `scripts/fetch_silero_vad.py` stages `silero_vad_16k_op15.onnx` (1,289,603 B) from a pinned commit of snakers4/silero-vad, sha256-checked, into the models pack beside the bundled voice, run beside `fetch_bundled_voice.py` by [`release-local.yml`](../../.github/workflows/release-local.yml) and the electron `package.json` scripts; the API and worker read it in place (step 4).

surfsense_local/electron/scripts/audiocpp/built-families.mjs

@@ -59,18 +59,42 @@ async function loadOutput(stage, work, family) {
   })
 }
 
-/** Throw unless the server in `stage` has a loader for every curated family. */
-export async function verifyBuiltFamilies(stage) {
+/**
+ * What one family's probe shows: `built`, its loader ran; `missing`, the build
+ * refused it by name; `unknown`, the server never got that far (blocked by
+ * antivirus, a library missing), which says nothing about the build.
+ */
+export function probeVerdict(family, output) {
+  if (loaderRan(family, output)) return "built"
+  if (output.includes(`unsupported model family hint: ${family}`)) return "missing"
+  return "unknown"
+}
+
+/** Each curated family the server in `stage` does not prove, with its verdict and output. */
+export async function unprovenFamilies(stage, { families = FAMILIES, load = loadOutput } = {}) {
   const work = mkdtempSync(join(tmpdir(), "surfsense-audiocpp-families-"))
   try {
     writeFileSync(join(work, "empty.gguf"), EMPTY_GGUF)
-    for (const family of FAMILIES) {
-      const output = await loadOutput(stage, work, family)
-      if (!loaderRan(family, output)) {
-        throw new Error(`audiocpp_server was not built with ${family}: ${output}`)
-      }
+    const unproven = []
+    for (const family of families) {
+      const output = await load(stage, work, family)
+      const verdict = probeVerdict(family, output)
+      if (verdict !== "built") unproven.push({ family, verdict, output })
     }
+    return unproven
   } finally {
     rmSync(work, { recursive: true, force: true })
   }
 }
+
+/** Throw unless the server in `stage` has a loader for every curated family. */
+export async function verifyBuiltFamilies(stage) {
+  const [first, ...rest] = await unprovenFamilies(stage)
+  if (!first) return
+  const families = [first, ...rest].map(({ family }) => family).join(", ")
+  throw new Error(
+    first.verdict === "missing"
+      ? `audiocpp_server was not built with ${families}: ${first.output}`
+      : `audiocpp_server did not run to prove ${families}: ${first.output || "it printed nothing"}`
+  )
+}

surfsense_local/electron/scripts/audiocpp/built-families.test.mjs

@@ -1,7 +1,7 @@
 import assert from "node:assert/strict"
 import test from "node:test"
 
-import { loaderRan } from "./built-families.mjs"
+import { loaderRan, probeVerdict, unprovenFamilies } from "./built-families.mjs"
 
 // What audiocpp_server v0.8.2 printed when started on an empty model package,
 // measured on Windows builds with and without each family.
@@ -21,3 +21,41 @@ test("a family the build left out is refused by name before any loader runs", ()
 test("another family's loader proves nothing about this one", () => {
   assert.equal(loaderRan("kokoro_tts", BUILT), false)
 })
+
+test("reads a family's probe as built, left out of the build, or never reached", () => {
+  assert.equal(probeVerdict("nemotron_asr", BUILT), "built")
+  assert.equal(probeVerdict("nemotron_asr", LEFT_OUT), "missing")
+  // A server that cannot start at all (blocked, a library missing) printed nothing
+  // about any family: it is not evidence of a build without them.
+  assert.equal(probeVerdict("nemotron_asr", ""), "unknown")
+})
+
+/** A stand-in for starting the server: what a build with only `built` prints. */
+function serverBuiltWith(built) {
+  return async (_stage, _work, family) =>
+    built.includes(family)
+      ? `audiocpp_server failed: failed to load model resources using builtin model spec for family '${family}' source 'gguf'`
+      : `audiocpp_server failed: unsupported model family hint: ${family}`
+}
+
+test("names every curated family a staged server was built without, with what it printed", async () => {
+  // A checkout staged before canary_asr joined the pins: the server says
+  // "unsupported model family hint: canary_asr" and speech cannot load.
+  const unproven = await unprovenFamilies("stage", {
+    families: ["kokoro_tts", "nemotron_asr", "canary_asr"],
+    load: serverBuiltWith(["kokoro_tts", "nemotron_asr"]),
+  })
+  assert.deepEqual(
+    unproven.map(({ family, verdict }) => [family, verdict]),
+    [["canary_asr", "missing"]]
+  )
+  assert.match(unproven[0].output, /unsupported model family hint: canary_asr/)
+})
+
+test("proves every family of a server built with all of them", async () => {
+  const unproven = await unprovenFamilies("stage", {
+    families: ["kokoro_tts", "canary_asr"],
+    load: serverBuiltWith(["kokoro_tts", "canary_asr"]),
+  })
+  assert.deepEqual(unproven, [])
+})

surfsense_local/electron/scripts/audiocpp/stage.mjs

@@ -19,6 +19,7 @@ import { packageManager } from "../not-staged/package-manager.mjs"
 import { renameWhenUnlocked } from "../windows-locks/rename-when-unlocked.mjs"
 import { TAG } from "./pins.mjs"
 import { copyServerFiles, SERVER } from "./server-files.mjs"
+import { stagedServer } from "./staged-server.mjs"
 import { unpackUpstream } from "./upstream-archive.mjs"
 
 const OUT = fileURLToPath(new URL("../../audiocpp", import.meta.url))
@@ -37,6 +38,18 @@ function verifyStage(stage) {
   }
 }
 
+/** Whether `folder` can be swapped out now; on Windows a running server holds it. */
+function canMove(folder, aside) {
+  try {
+    renameSync(folder, aside)
+  } catch (error) {
+    if (["EBUSY", "EPERM", "EACCES"].includes(error.code)) return false
+    throw error
+  }
+  renameSync(aside, folder)
+  return true
+}
+
 function megabytes(root) {
   let total = 0
   for (const entry of readdirSync(root, { withFileTypes: true, recursive: true })) {
@@ -47,10 +60,18 @@ function megabytes(root) {
 
 async function main() {
   const strict = process.argv.includes("--strict")
-  if (existsSync(join(OUT, SERVER))) {
+  const staged = await stagedServer(OUT)
+  if (staged.state === "current") {
     console.log(`audio.cpp ${TAG} already staged in ${OUT}`)
     return
   }
+  // Without --strict a stale server is kept whenever staging again cannot finish:
+  // it still serves every family it has, and pnpm dev goes on.
+  const keep = (why) => {
+    const message = `${why}\n\nThe audio.cpp in ${OUT} stays, though it ${staged.reason}.`
+    if (strict) throw new NotStaged(message)
+    console.warn(message)
+  }
 
   const host = `${process.platform}-${process.arch}`
   const reason = !HOSTS.includes(host)
@@ -64,6 +85,7 @@ async function main() {
           packageManager()
         )
   if (reason) {
+    if (staged.state === "stale") return keep(reason)
     if (strict) throw new NotStaged(reason)
     // Empty, so the packager still finds the folder; the sidecar never starts.
     mkdirSync(OUT, { recursive: true })
@@ -71,10 +93,17 @@ async function main() {
     return
   }
 
+  if (staged.state === "stale") console.log(`audio.cpp in ${OUT} ${staged.reason}; staging it again`)
+
   const started = Date.now()
   const work = mkdtempSync(join(tmpdir(), "surfsense-audiocpp-"))
   const stage = `${OUT}.stage-${process.pid}`
   const backup = `${OUT}.old-${process.pid}`
+  // Found out before a few minutes of compiling, not after.
+  if (staged.state === "stale" && !canMove(OUT, backup)) {
+    rmSync(work, { recursive: true, force: true })
+    return keep(`${OUT} is in use; quit SurfSense and run this again.`)
+  }
   try {
     mkdirSync(stage)
     if (process.platform !== "darwin") {
@@ -98,6 +127,9 @@ async function main() {
     rmSync(backup, { recursive: true, force: true })
     const minutes = ((Date.now() - started) / 60_000).toFixed(1)
     console.log(`audio.cpp ${TAG} staged in ${OUT}: ${megabytes(OUT)} MB, in ${minutes} min`)
+  } catch (error) {
+    if (staged.state !== "stale") throw error
+    keep(`audio.cpp could not be staged again: ${error instanceof Error ? error.message : error}`)
   } finally {
     rmSync(work, { recursive: true, force: true })
     rmSync(stage, { recursive: true, force: true })

surfsense_local/electron/scripts/audiocpp/staged-server.mjs

@@ -0,0 +1,19 @@
+// Whether electron/audiocpp holds a server to keep, one to stage again, or none.
+// A server staged before a family joined the pins answers that family's model
+// with "unsupported model family hint", so it is not kept as it is.
+import { existsSync } from "node:fs"
+import { join } from "node:path"
+
+import { unprovenFamilies } from "./built-families.mjs"
+import { SERVER } from "./server-files.mjs"
+
+/** `absent`, `current`, or `stale` with the reason, as a log line's end. */
+export async function stagedServer(out, { exists = existsSync, probe = unprovenFamilies } = {}) {
+  if (!exists(join(out, SERVER))) return { state: "absent" }
+  const unproven = await probe(out)
+  if (unproven.length === 0) return { state: "current" }
+  const missing = unproven.filter(({ verdict }) => verdict === "missing").map(({ family }) => family)
+  if (missing.length) return { state: "stale", reason: `was built without ${missing.join(", ")}` }
+  const output = unproven[0].output.trim().slice(0, 400)
+  return { state: "stale", reason: `did not run (${output || "it printed nothing"})` }
+}

surfsense_local/electron/scripts/audiocpp/staged-server.test.mjs

@@ -0,0 +1,40 @@
+import assert from "node:assert/strict"
+import test from "node:test"
+
+import { stagedServer } from "./staged-server.mjs"
+
+const nothingUnproven = async () => []
+
+test("with no server staged, staging starts from nothing", async () => {
+  const staged = await stagedServer("out", { exists: () => false, probe: nothingUnproven })
+  assert.equal(staged.state, "absent")
+})
+
+test("keeps a staged server that proves every curated family", async () => {
+  const staged = await stagedServer("out", { exists: () => true, probe: nothingUnproven })
+  assert.equal(staged.state, "current")
+})
+
+test("stages again a server built before a family joined the pins, naming it", async () => {
+  const staged = await stagedServer("out", {
+    exists: () => true,
+    probe: async () => [
+      { family: "canary_asr", verdict: "missing", output: "unsupported model family hint: canary_asr" },
+    ],
+  })
+  assert.equal(staged.state, "stale")
+  assert.match(staged.reason, /built without canary_asr/)
+})
+
+test("says a staged server that would not run did not run, not that it lacks families", async () => {
+  const staged = await stagedServer("out", {
+    exists: () => true,
+    probe: async () => [
+      { family: "kokoro_tts", verdict: "unknown", output: "" },
+      { family: "canary_asr", verdict: "unknown", output: "" },
+    ],
+  })
+  assert.equal(staged.state, "stale")
+  assert.match(staged.reason, /did not run/)
+  assert.doesNotMatch(staged.reason, /built without/)
+})