qwen-code/packages/web-shell/client/voice/voiceModels.ts
qqqys d360dd276e
feat(serve): voice dictation over the daemon for the Web Shell (#5755)
* feat(serve): voice dictation over the daemon for the Web Shell

Bring voice input to the `qwen serve` Web Shell. The browser captures the
microphone, streams raw 16 kHz mono PCM to a new `/voice/stream` WebSocket, and
the daemon transcribes server-side by reusing the existing CLI voice pipeline
(realtime streaming + on-stop batch) — provider credentials never reach the
browser, and it works whether the daemon is local or remote.

Daemon:
- `/voice/stream` WebSocket handler (serve/voice) reusing openVoiceStream /
  openQwenAsrRealtimeStream / transcribeVoiceAudio; resolves the workspace
  voiceModel from a ModelsConfig built off settings.
- Routed through the existing ACP upgrade listener so it shares the
  loopback / host-allowlist / CSRF / bearer checks; concurrency-capped.
- Advertises the `voice_transcribe` capability (unconditional, like
  auth_device_flow; the WS errors when no voice model is configured).
- Relax `Permissions-Policy` to `microphone=(self)` so the same-origin shell can
  request the mic — an empty `microphone=()` allowlist blocked the prompt
  entirely.
- Allowlist `voiceModel` on `/workspace/settings` so the picker can read it.

Web Shell:
- Mic button in the composer: click to record, click to stop -> the transcript
  is inserted into the input for review before sending.
- `/model --voice` picker (sourced from `/workspace/providers`, since voice
  models are hidden from the session model list) and `/model --voice <id>`,
  persisted via the prompt channel like `/model --fast`.

The voiceModel resolver now accepts a structural model lookup so the daemon can
resolve it without constructing a full CLI Config.

Co-authored-by: Qwen-Coder <noreply@qwen.ai>

* fix(serve): harden web voice streaming

* fix(serve): clean stale voice capture resources

* fix(serve): harden voice websocket errors

* fix(serve): hide voice capability when token auth is configured

* fix(serve): address voice review blockers

* fix(web-shell): keep voice cancel available when disabled

* fix(web-shell): inline the voice mic in the composer toolbar

The mic was an absolute-positioned overlay anchored to the composer
wrapper, so after the #5775 chat-UI restructure it floated outside the
input box (clipped at the bottom-right). Render it inside ChatEditor's
`toolbarRight`, just before the send button, wiring the transcript insert
to the composer core — it sits inline with the `/`, `@`, and send
controls like the rest of the toolbar.

Restyle to a Codex-style recording experience: an idle mic that matches
the other toolbar icon buttons, and a recording pill with a live waveform
(driven by the RMS meter), an elapsed `M:SS` timer, and a stop control.
Also forward `/voice/stream` through the vite dev proxy (`ws: true`,
scoped to the exact path so it doesn't shadow the client's own
`client/voice/*` source modules) so the WebSocket reaches the daemon in
dev instead of silently failing against vite.

Co-authored-by: Qwen-Coder <noreply@qwen.ai>

* feat(serve): authenticate the voice WebSocket via a bearer subprotocol

Voice was suppressed on any token-configured daemon because browsers
cannot set an `Authorization` header on a WebSocket, so the bearer check
on the shared ACP upgrade listener always failed for the browser mic.

Let the browser carry the token in `Sec-WebSocket-Protocol` as
`qwen-bearer.<base64url(token)>` (the only header a browser can set on a
WS). The upgrade listener now accepts the token from either the
`Authorization` header (non-browser clients, unchanged) or the
subprotocol, hash-compared in constant time. `handleProtocols` selects a
non-secret subprotocol (or none) so the token is never echoed back in the
handshake response. This aligns the voice WS with how the ACP WS will
authenticate browsers.

With the WS now authenticated, drop the `!tokenConfigured` /
`requireAuth` suppression of the `voice_transcribe` capability — it is
advertised whenever the `/voice/stream` endpoint exists. Tests updated to
assert voice is advertised under a token / `--require-auth`.

Co-authored-by: Qwen-Coder <noreply@qwen.ai>

* fix(serve): make the voice WS subprotocol handshake complete + test it

Review follow-up to the bearer-subprotocol auth. Fixes a real handshake
bug and closes the test gap on the security-critical path.

- Handshake bug: when the browser offered only the `qwen-bearer.*`
  subprotocol, the daemon's `handleProtocols` selected none, and a strict
  WS client (the `ws` library) aborts with "Server sent no subprotocol".
  The web-shell now offers a non-secret marker (`qwen-ws`) alongside the
  bearer one, so the daemon always selects the marker — never echoing the
  secret — and the handshake completes for strict and lenient clients
  alike.
- Tests: add daemon-side coverage for the subprotocol auth path (accept
  valid token, reject wrong/malformed, no-token loopback) and assert the
  secret is never echoed back; add client coverage that the bearer token
  is offered as `[qwen-ws, qwen-bearer.<b64url>]` and round-trips.
- Drop the unreachable try/catch around `Buffer.from(_, 'base64url')`
  (it never throws) and correct the comment.
- a11y: mark the transcribing/interim regions as `role="status"` /
  `aria-live`, and add a `:focus-visible` ring to the idle mic.
- Refresh the stale `useVoiceCapture` doc comment (token deployments are
  now supported in the browser) and fix the mock typing in
  VoiceButton.test.tsx.

Co-authored-by: Qwen-Coder <noreply@qwen.ai>

* fix(web-shell): ignore stale voice socket callbacks

* test(serve): sync voice capability integration expectation

* fix(serve): harden voice websocket review gaps

* fix(serve): sanitize voice websocket failures

* fix(serve): address voice review suggestions

* fix(serve): harden voice websocket lifecycle

---------

Co-authored-by: Qwen-Coder <noreply@qwen.ai>
2026-06-24 11:32:43 +08:00

76 lines
2.2 KiB
TypeScript

/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
/** A voice-transcription model option for the `/model --voice` picker. */
export interface VoiceModelOption {
/** Raw model id (no auth suffix) — what gets persisted as `voiceModel`. */
id: string;
label?: string;
authType?: string;
baseUrl?: string;
contextWindow?: number;
modalities?: { audio?: boolean };
}
/**
* Mirror of the CLI's `resolveVoiceTransport` id patterns (voice-model.ts): true
* for ids the daemon has an ASR transport for. Kept in sync by hand because the
* Web Shell can't import the CLI's voice modules.
*/
export function isVoiceModelId(id: string): boolean {
const s = id.toLowerCase();
return (
/^qwen3-asr-flash-realtime(?:-|$)/.test(s) ||
/^qwen3-asr-flash(?:-\d{4}-\d{2}-\d{2})?$/.test(s) ||
/^(fun-asr|paraformer).*realtime(?:-|$)/.test(s)
);
}
interface ProvidersStatusLike {
providers?: Array<{
authType?: string;
models?: Array<{
baseModelId?: string;
modelId?: string;
name?: string;
baseUrl?: string;
contextLimit?: number;
isRuntime?: boolean;
}>;
}>;
}
/**
* Extract selectable voice models from a `/workspace/providers` status. Voice
* models are hidden from the session's main model list (`voiceOnly`), so the
* picker sources them from the providers surface instead.
*/
export function extractVoiceModels(
status: ProvidersStatusLike | undefined,
): VoiceModelOption[] {
const seen = new Set<string>();
const out: VoiceModelOption[] = [];
for (const provider of status?.providers ?? []) {
for (const model of provider.models ?? []) {
const id = model.baseModelId;
if (!id || model.isRuntime || !isVoiceModelId(id) || seen.has(id)) {
continue;
}
seen.add(id);
out.push({
id,
...(model.name ? { label: model.name } : {}),
...(provider.authType ? { authType: provider.authType } : {}),
...(model.baseUrl ? { baseUrl: model.baseUrl } : {}),
...(typeof model.contextLimit === 'number'
? { contextWindow: model.contextLimit }
: {}),
modalities: { audio: true },
});
}
}
return out;
}