* feat(desktop): voice dictation in the desktop app
Bring the `/voice` dictation feature to the desktop app (#5796), matching the
Web Shell (#5755). The renderer captures the microphone and streams raw 16 kHz
mono PCM to a loopback `/voice/stream` WebSocket in the Electron main process;
the server transcribes by reusing the CLI voice pipeline (batch + realtime) with
the qwen credentials, so provider keys never reach the renderer.
- server-core/voice: a standalone loopback WS server (separate from the RPC
server, shares its token via the `?token=` query) + ported batch/realtime
transcription pipeline + SSRF guard.
- Credentials resolve from the qwen CLI config: OAuth (`~/.qwen/oauth_creds.json`)
→ API key (`~/.qwen/settings.json` DashScope provider) → env.
- Renderer: a mic in the composer toolbar that swaps to a recording bar
(waveform + timer) while dictating; transcript is inserted for review.
- Voice on/off and model are configurable in Settings → Input and the composer
model dropdown, persisted via a `voiceEnabled` / `voiceModel` setting.
Co-authored-by: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): harden voice dictation review gaps
* test(desktop): update voice ipc channel inventory
* fix(desktop): hide voice stream internals from errors
* test(desktop): cover voice websocket paths
* test(desktop): cover voice transcription paths
* fix(desktop): detect voice capture frame drops
* fix(desktop): address voice review blockers
* fix(desktop): address voice security blockers
* fix(desktop): address voice review blockers
* fix(desktop): address voice review blockers
* fix(desktop): address voice shutdown review
* fix(desktop): address voice review blockers
* fix(desktop): fix server-core voice typecheck and harden config resolution
Fix three type errors that left server-core's typecheck red:
- type the readQwenJson test mocks generically (`<T,>() => ... as T | undefined`)
to match the real `<T>(file) => Promise<T | undefined>` signature
- replace `HeadersInit` (undefined under lib ESNext/no DOM) with
`RequestInit['headers']`
- widen the fetch stub cast to `as unknown as typeof fetch` (the stub lacks
`preconnect`)
Also apply small hardening to resolve-voice-config:
- normalizeBaseUrl strips userinfo so a user:pass@host base URL can't leak
credentials into outbound requests or errors
- OPENAI_API_KEY without OPENAI_BASE_URL now throws a specific message instead
of a generic no-credentials error
- readQwenJsonFromDisk honors QWEN_HOME for the config base dir
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): address voice review follow-ups (logging, dedupe, truncation)
- voice-ws-handler: tag connection logs with a short per-connection id so
concurrent sessions are distinguishable; log unrecognized text frames; add
bufferedBytes/pcmChunks context to the finalize "no session/audio" warnings.
- voice-frame-sender: fire onTooManyDroppedFrames once as the threshold is
first crossed instead of on every later dropped frame.
- qwen-asr-realtime-session: rename `failed` -> `settled` for parity with
voice-stream-session (the flag is set on both fail and the success path).
- voice-stream-session / qwen-asr-realtime-session: drop the redundant outer
slice in formatServerErrorMessage so sanitize's truncation `...` indicator
survives (credential redaction unchanged).
- voice-server: drop the dead `if (!isEnabled())` wrapper that the early
return above already makes unconditional.
- voice-stream-session test: cover the result-generated sentence_end commit
branch and the lastPartial reset (multi-sentence path).
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): normalize QWEN_HOME and scope voice pending-op cap to control frames
Voice config resolution read the raw QWEN_HOME env value, so QWEN_HOME=~/x or
a relative value pointed desktop voice at a different directory than the rest
of Qwen (which normalizes via core's Storage.getGlobalQwenDir) and missed the
OAuth/settings credentials. Mirror core's normalization in a local helper:
expand a leading ~, resolve a relative path to absolute, and fall back to
~/.qwen when QWEN_HOME is unset/empty.
Also scope the per-connection pending-operation cap to control frames. PCM
frames buffered behind a slow upstream connect (up to the 8s connect timeout)
no longer inflate the count cap and falsely trip "Too many pending voice
messages." — buffered audio is already bounded by queuedBytes. The drain gate
(pendingOperations) still counts every message, so close/slot-release
semantics are unchanged.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): reject credentialed voice base URLs; scope batch byte cap to batch
Address qwen-code-ci-bot review on the desktop voice ASR handler.
- resolve-voice-config: normalizeBaseUrl now throws on embedded userinfo
instead of stripping it. For `https://real-host@evil.com/...` the URL
parser already resolves `evil.com` as the host, so clearing userinfo
afterward would silently proceed with an attacker-controlled host. A
legitimate voice base URL never carries credentials.
- voice-ws-handler: MAX_AUDIO_BYTES (~5.5-min / 10 MB) is the batch WAV/file
cap. Streaming forwards frames immediately and is bounded by
MAX_CONNECTION_MS (the 6-min hard timer) + queuedBytes, so counting
already-forwarded frames toward the batch cap cut legit streams off ~30 s
early. Enforce the file cap for batch sessions only; the batch 10 MB cap
is unchanged.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): redact voice stream errors; drop unused qwen:// origin; re-fire voice URL retry
Address voice review on 8b19cffcf:
- Streaming session errors reached the renderer raw: the onError callback and
the streaming open/finish paths forwarded errMessage(error) straight to
fail(), bypassing the credential redaction the batch path applies. An
upstream ws.on('error') can surface auth URLs / Bearer tokens, so route
streaming error text through the same sanitizeResponseDetails redactor
(passing the apiKey) before it leaves the server. Batch errors are already
sanitized inside transcribeQwenAsrBatch and stay untouched.
- Remove the qwen:// scheme from the voice origin allowlist. No custom app
scheme is registered anywhere (only craftagents deep-links and the
thumbnail:// privileged scheme), so accepting it only let an unregistered
same-machine scheme pass origin validation — tightened as defense-in-depth.
- Fix the voice URL cold-start retry: setWsUrl(null) is a no-op when the value
stays null, so the [wsUrl] effect never re-fired and voice stayed unavailable
on slow cold starts (OAuth refresh / slow disk > 1.5s). Add a bounded retry
counter to the effect deps so it re-polls until the URL resolves.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): address voice PR #5856 review feedback
Cleanups:
- voice-stream-session: set settled in the close-handler else branch so a late
error/close cannot re-enter and double-fire onError.
- qwen-asr-realtime-session: extract the duplicated "connection closed
unexpectedly" message into a single module constant.
- shared/storage: inline the no-op loadStoredConfigForVoiceSettings wrapper.
Tests:
- voice-stream-retry: cover openVoiceStreamWithRetry (first-try success,
retry-then-success, non-retryable rethrow, give-up) using fake timers.
- voice-server: extract pure classifyVoiceUpgrade for the 4 upgrade guards;
unit-test the exact statuses and add an integration test asserting each
guard rejects without upgrading.
- net-guard: add a DNS-lookup-failure test for assertVoiceBaseUrlNetworkAllowed.
Fix:
- useVoiceCapture: treat a graceful 1000 close as normal completion instead of
surfacing it as a connection error.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): order voice upgrade token check before disk read; settle realtime close
Address two voice PR #5856 review notes:
- classifyVoiceUpgrade: check the voice token before the disk-reading
isEnabled guard, so an upgrade with a wrong/missing token is rejected
without a loadStoredConfig() disk read (a pre-auth wasted-work surface).
New guard order: path -> token -> disabled -> origin.
- qwen-asr-realtime-session: set settled=true in the ws 'close' handler's
terminal branches, mirroring voice-stream-session, so a late error/close
event can't re-enter via fail() and double-fire reject/onError.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): harden voice backpressure visibility + IPC sender guard (PR #5856)
Criticals (wenshao):
- voice-stream-session: count dropped audio frames/bytes under upstream
(DashScope) backpressure and log the cumulative total on entry and on
recovery. Previously frames were dropped silently, leaving invisible
transcript gaps with no log/counter.
- electron main: validate the sender of `__get-voice-stream-url`. The URL
embeds the loopback auth token, so only the app's own top-level renderer
frame (file:// when packaged, the Vite dev origin in dev) may read it;
injected/cross-origin sub-frames and stray webviews get null.
Suggestions (wenshao):
- InputSettingsPage: revert the optimistic voice enabled/model state when the
IPC write rejects, so the UI never lies about the persisted value.
- useVoiceDictation: tick the elapsed timer at 1s instead of 100ms — the bar
floors elapsed to whole seconds, so finer ticks were ~10x redundant renders.
- voiceModels: add `description` as the single source of truth and consume it
in both the settings and composer pickers (drops duplicated computation).
- voice-server: gracefully WS-close clients before force-terminating on
shutdown, so an in-flight transcript can flush instead of being dropped by a
TCP reset; bounded by a short grace period and an absolute deadline.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): await + rollback voice model change on IPC failure
handleVoiceModelChange in FreeFormInput fired the setVoiceModel IPC and
discarded the promise (void). If that IPC rejected (handler throws,
settings file locked), the renderer kept the optimistically-selected
model while the persisted value stayed old. The voice server reads the
persisted model per recording (resolveDesktopVoiceConfig), so the
dropdown and the actual ASR model silently diverged.
Await the IPC and roll back the optimistic setVoiceModel on failure,
matching the established pattern in InputSettingsPage. Failures are
reported through the existing logInputError helper.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): report voice backpressure totals at session end; log graceful shutdown
Address ci-bot review on the desktop voice PR.
- voice-stream-session: surface the cumulative dropped-frame total exactly once
when a session ends. droppedFrames/droppedBytes were warned only on
backpressure enter/recover transitions, so a session that ended while still
dropping (or after a recovery) lost the running total. A one-shot
reportDroppedTotals() now fires from every terminal path (fail, task-finished,
unexpected close), logging only when frames were actually dropped. Adds tests
that a dropping session logs the total once and a clean session logs nothing
(mutation-checked: disabling the report fails the test).
- voice-server: closeVoiceServerResources was silent during graceful shutdown,
unlike the disabled-close path. Thread the existing Logger in and add concise
logs matching the existing style: shutdown start with active client count,
force-terminate of stragglers after the grace period, and shutdown complete.
terminateVoiceClients now returns the count it terminated so the
force-terminate log is precise. Adds a test asserting all three lines.
The reviewer's mock.module ordering concern for the backpressure test was
assessed and found non-fragile: the test registers the platform mock and then
dynamically imports the module under test (await import after mock.module), so
bun re-instantiates voice-stream-session.ts with the mock active regardless of
load order. Verified across isolation, the full voice suite, and orderings that
pre-load the module with the real logger first; all green. No refactor made.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* test(desktop): cover voice dropped-totals fail/close paths; clarify recovery-log comment
reportDroppedTotals() runs from three terminal paths but only the
task-finished path was tested. Add deterministic tests for the two
untested paths and lock the idempotency guard:
- fail() path: an upstream socket error drives fail() (which sets
`settled` before reporting); assert the totals line logs exactly once.
- ws.on('close') path: a mid-session close (which reports before setting
`settled`, the inverse order of fail()); assert it logs exactly once.
- idempotency: a close followed by a late task-finished invokes
reportDroppedTotals() twice — task-finished is not short-circuited by
`settled` — so only the droppedTotalsReported guard keeps the line to a
single entry. Mutation-verified: removing the guard double-logs;
disabling the fail()-path report drops the fail() test's log.
Also reword the backpressure-recovery comment: droppedFrames/droppedBytes
are cumulative session counters (never reset between episodes), so the
recovery log reports the running total, not a per-episode count.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): log disabled-path straggler terminations; drop redundant default args
On the voice stream server, the disabled-grace close path force-terminated
straggling clients with no diagnostic, unlike the shutdown path which logs
the force-terminated count. Add a parity warn via a small exported
terminateDisabledVoiceClients() helper (a clean unit-test seam, matching the
existing exported close/terminate helpers) and cover it with a deterministic
test.
Also stop hard-coding CLOSE_TIMEOUT_MS / SHUTDOWN_GRACE_MS at the
closeVoiceServerResources() call site: they were only the function's own
defaults, forwarded just to reach the log param, and would silently drift if
the defaults changed. Pass undefined so the defaults apply.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
* fix(desktop): count realtime ASR dropped frames; cover frame-trust gate; sanitize status
- voice(server-core): realtime ASR session counts dropped frames/bytes under
socket backpressure and logs the cumulative total once at session end
(parity with voice-stream-session reportDroppedTotals; idempotent across the
fail/session.finished/close terminal paths). Logger is injectable via deps so
the drop totals are unit-tested deterministically.
- voice(electron): surface voice-server URL retry exhaustion instead of silently
disabling dictation — one-shot console.warn after the retry budget plus a new
initFailed flag on the hook return. Retry timing/behavior unchanged.
- security(electron): extract isTrustedRendererFrameUrl (the sole guard gating
the voice token) into main/voice/frame-trust.ts and unit-test file://, dev
origin, cross-origin, undefined, malformed, and the VITE_DEV_SERVER_URL
fallback. Behavior preserved; index.ts imports the helper.
- voice(server-core): sanitize response.statusText with sanitizeResponseDetails
like the body, so a hostile ASR proxy can't leak secrets via the status line.
Co-Authored-By: Qwen-Coder <noreply@qwen.ai>
---------
Co-authored-by: Qwen-Coder <noreply@qwen.ai>
Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com>
a1bcda6c1b