talkis/docs/audio-pipeline-principles.md

Audio Pipeline Principles

This document is the working contract for Talkis audio changes. Future agents should update it when the audio architecture changes.

Goals

• Keep ordinary voice dictation fast.
• Keep transcription quality stable.
• Avoid moving expensive conversion into hot paths.
• Keep local, cloud, file, and call behavior explicit instead of relying on hidden fallbacks.
• Preserve enough logs to diagnose audio quality and runtime behavior from ~/.talkis/talkis.log.

Pipeline Map

Voice dictation:

Widget hotkey/button
-> useWidgetRecording.ts
-> recordingRuntime.ts
-> native_voice_recorder.rs first
-> processRecordingBlob()
-> ai::transcribe_and_clean()
-> paste_text

File transcription:

Files tab / widget file drop
-> src/lib/fileTranscription.ts
-> ai::transcribe_file_path()
-> media.rs preparation/chunking
-> local, custom, or cloud STT

Call transcription:

Widget call mode
-> src/lib/callCapture.ts
-> call_capture.rs for system track
-> recordingRuntime.ts for mic track
-> transcribeCallCaptureSession()
-> file transcription pipeline

Realtime Interpreter:

Settings / Realtime Interpreter beta
-> src/lib/realtimeInterpreter.ts
-> realtime_interpreter.rs
-> real mic + system audio adapters
-> Talkis Cloud WebSocket proxy
-> Gemini Live Translate sessions
-> virtual mic output + local playback

Voice Dictation

The primary voice path is native Rust capture:

• src-tauri/src/native_voice_recorder.rs
• Tauri commands:
  • start_native_voice_recording
  • pause_native_voice_recording
  • resume_native_voice_recording
  • stop_native_voice_recording
• Output contract:
  • audio_base64
  • mime_type: "audio/wav"
  • file_name: "recording.wav"
  • duration_ms
  • sample_rate: 16000
  • channels: 1
  • peak
  • rms

The recorder uses cpal, stores microphone samples in memory, converts to mono, resamples to 16 kHz, writes PCM16 WAV, and logs stats on stop.

Important implementation detail: on macOS, cpal::Stream is not safe to keep in a global static. Keep the stream alive on its own recorder thread and store only thread-safe control handles in global state.

Voice Fallback

Keep WebView MediaRecorder fallback in recordingRuntime.ts.

Fallback is required when:

• native recorder fails to start;
• a selected microphone exists, but only WebView deviceId can identify it reliably;
• platform-specific microphone permissions or device routing behave differently than cpal.

Do not remove the fallback unless selected-microphone parity is proven on macOS, Windows, and Linux.

Local STT Input Format

Managed local Whisper expects:

WAV, 16 kHz, mono, PCM 16-bit

Rules:

• Native voice recording should already produce this format.
• media::convert_audio_to_local_stt_wav() must skip ffmpeg when input is already ready.
• File transcription should also skip ffmpeg for ready WAV files that fit into one STT request.
• ffmpeg remains the correct path for arbitrary audio/video, WebM/Opus, MP3/M4A/MP4, diarization prep, and chunking.

When editing conversion code, preserve the logs:

• Running bundled ffmpeg sidecar
• Bundled ffmpeg sidecar finished in ...ms
• System ffmpeg fallback finished in ...ms
• Skipping ffmpeg for local STT...
• Skipping ffmpeg for file transcription...

Local Whisper Hallucination Guardrails

Long local Whisper jobs can produce repeated caption-like text on silence, for example:

• Спасибо. Спасибо. Спасибо.
• Продолжение следует...
• repeated copies of the last real phrase

This is not a UI recursion bug. It usually means Whisper received a long low-signal or silent region and reused context across internal windows.

Preserve these safeguards:

• src-tauri/src/bin/talkis-stt.rs
  • params.set_no_context(true)
  • params.set_suppress_nst(true)
  • low temperature / no temperature increment
  • entropy threshold
• src-tauri/src/ai.rs
  • known hallucination detection
  • repetitive transcript text sanitizer
  • repetitive timestamped segment filter before diarization assembly

If changing these filters, test against:

• short real voice dictation;
• long meeting/call recording with pauses;
• mostly silent audio;
• file transcription with speaker diarization enabled.

File Transcription

File transcription is path-based. Do not load large files into WebView memory.

Rules:

• Keep src/lib/fileTranscription.ts as the frontend entry point.
• Keep native path invocation through transcribe_file_path.
• Keep chunk progress events and per-chunk logs.
• Use ffmpeg for video and unsupported formats.
• For local mode, each chunk must be converted to the local STT WAV contract before hitting the runtime.
• For cloud mode, use the existing proxy endpoints and do not silently switch to local diarization.

Chunking currently protects API limits and long recordings. Do not remove chunking unless the target endpoint is proven to handle the full file size and duration.

Call Capture

Call capture has two different tracks:

• mic track: user microphone, handled through the existing recording runtime / file pipeline;
• system track: platform-specific system audio capture.

Current system-audio support:

• macOS: implemented via Core Audio process tap / aggregate device in call_capture.rs;
• Windows: implemented via cpal WASAPI loopback on the default output device in call_capture.rs;
• Linux: implemented via PipeWire default output monitor capture in call_capture.rs.

Linux call system capture is PipeWire-only. PulseAudio-only systems without PipeWire remain unsupported and should show an honest PipeWire unavailable / monitor not found error instead of silently falling back.

For macOS, Windows, and Linux system track diagnostics, rely on stop-time logs:

System audio capture level: max=... dBFS, frames_above_noise_floor=...

If max=-120.0 dBFS and frames_above_noise_floor=0, the system track is silent. The transcript should not treat that as usable remote-speaker audio.

Realtime Interpreter

Realtime Interpreter is a separate beta audio path. It must not be wired into transcribeCallCaptureSession(). Realtime translation history is allowed only from final live transcript segments collected by the interpreter session itself.

Current contract:

• Tauri commands live in src-tauri/src/realtime_interpreter.rs.
• Frontend command wrappers live in src/lib/realtimeInterpreter.ts.
• Settings UI lives in src/windows/settings/tabs/RealtimeInterpreterTab.tsx.
• V1 is cloud-only. The desktop must use the saved Talkis deviceToken; do not expose Google API keys, BYOK, or direct Google WebSocket credentials in the UI.
• Desktop defaults to wss://proxy.talkis.ru/api/realtime-translate.
• The proxy is responsible for JWT/subscription checks and for bridging to Gemini Live Translate with gemini-3.5-live-translate-preview.
• Gemini Live Translate input is raw PCM16, 16 kHz, mono, little-endian, sent as audio/pcm;rate=16000. Output audio is raw PCM16, 24 kHz, mono, returned as audio/pcm;rate=24000.
• Transcript events should carry both source and translated text from inputAudioTranscription and outputAudioTranscription.
• External virtual audio drivers are required for MVP routing: BlackHole on macOS, VB-CABLE on Windows, PipeWire virtual source/sink on Linux.
• Supported v1 language pairs keep Russian as the local user language: RU -> EN / EN -> RU, RU -> ES / ES -> RU, RU -> DE / DE -> RU, and RU -> zh-Hans / zh-Hans -> RU.
• Start requests must validate a virtual mic output, Talkis Cloud credentials, system-audio support, and a headphones/separate-output confirmation before any streaming begins.
• test_virtual_mic_output may play a short test tone to the selected virtual output; it must not silently fall back to the real speakers.
• On stop, if final transcript segments exist, save a call-like history entry: source: "call", fileName: "Перевод звонка", callSessionId, duration, status, original text in raw, bilingual translated text in cleaned, and optional translation metadata with provider/model/language pair/segments.
• Do not retry realtime translation from history unless raw audio tracks are explicitly saved in a later feature.

Do not present realtime translation as fully running until the Talkis proxy WebSocket transport and platform audio routing are implemented and verified.

Speaker Diarization

Local speaker diarization uses:

• local Whisper segments with timestamps;
• local diarization runtime segments;
• overlap/nearest matching in ai.rs;
• final formatting in format_speaker_transcript().

Rules:

• Do not assemble speaker transcripts from STT text without timestamps.
• Filter known repeated/hallucinated STT segments before assigning speakers.
• If the system track has no diarizable speech, the existing mic fallback is acceptable, but logs must make that explicit.
• Speaker labels shown to users should stay product-facing: Вы, Гость N.

Logging Contract

Audio bugs are usually runtime bugs, not static type bugs. Keep logs specific.

Required evidence:

• recorder path used: native WAV, WebView WAV, WebM, or fallback;
• selected mic and active device label when available;
• audio stats: duration, sample rate, channels, peak, RMS;
• ffmpeg start and finish timing;
• STT endpoint and response status;
• file chunk index, total chunks, and chunk size;
• call system capture level and source/stored format.

Do not log API keys, device tokens, or full local model paths if they include sensitive user names.

Verification

Minimum checks after audio pipeline edits:

bunx tsc --noEmit
cargo check --manifest-path src-tauri/Cargo.toml
git diff --check

For Rust logic in ai.rs, run targeted tests:

cargo test --manifest-path src-tauri/Cargo.toml ai::tests --lib

Manual checks when behavior changes:

• macOS voice dictation recognizes a real phrase and logs native recorder stats.
• Local voice dictation does not run bundled ffmpeg when native WAV is used.
• WebView fallback still records when native capture fails or selected mic cannot be mapped.
• Ready 16 kHz mono PCM WAV file skips ffmpeg.
• MP3/MP4/WebM files still go through ffmpeg.
• Long local call/file transcription with pauses does not produce repeated Спасибо / Продолжение следует.
• macOS call capture does not regress; Windows call capture writes a non-empty system.wav; Linux PipeWire call capture writes a non-empty system.wav; Linux without PipeWire remains clearly unsupported.

Release Notes For Audio Dependencies

cpal pulls platform audio backends. Linux call system capture also depends on PipeWire.

Linux release jobs need libasound2-dev for ALSA builds and libpipewire-0.3-dev for PipeWire system-audio capture. If changing cpal features, PipeWire bindings, or replacing the recorder backend, re-check .github/workflows/release.yml and Linux Tauri dependency installation.