mirror of
https://github.com/SerTimBerrners-Lee/talkis.git
synced 2026-07-09 17:29:15 +00:00
Prepare release v0.1.22
This commit is contained in:
David Perov
parent
c3cf031c21
commit
98f00a4bd1
17 changed files with 1608 additions and 34 deletions
27
.github/workflows/release.yml
vendored
27
.github/workflows/release.yml
vendored
|
|
@ -59,6 +59,7 @@ jobs:
|
|||
libwebkit2gtk-4.1-dev \
|
||||
libayatana-appindicator3-dev \
|
||||
libxdo-dev \
|
||||
libasound2-dev \
|
||||
librsvg2-dev \
|
||||
patchelf \
|
||||
clang \
|
||||
|
|
@ -146,6 +147,32 @@ jobs:
|
|||
BUILD_ROOT: release-artifacts
|
||||
TALKIS_REQUIRED_UPDATER_PLATFORMS: darwin-aarch64,windows-x86_64,linux-x86_64
|
||||
|
||||
- name: Prepare stable download assets
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
STABLE_DIR="release-artifacts/stable-downloads"
|
||||
mkdir -p "$STABLE_DIR"
|
||||
|
||||
copy_first() {
|
||||
local pattern="$1"
|
||||
local target="$2"
|
||||
local source
|
||||
|
||||
source="$(find release-artifacts -type f -name "$pattern" ! -path "$STABLE_DIR/*" -print -quit)"
|
||||
if [[ -z "$source" ]]; then
|
||||
echo "Missing release asset matching $pattern" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cp "$source" "$STABLE_DIR/$target"
|
||||
echo "Prepared $target from $source"
|
||||
}
|
||||
|
||||
copy_first "*.dmg" "Talkis-macos.dmg"
|
||||
copy_first "*.exe" "Talkis-windows-x64-setup.exe"
|
||||
copy_first "*.AppImage" "Talkis-linux-x64.AppImage"
|
||||
|
||||
- name: Publish GitHub release assets
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
|
|
|
|||
24
AGENTS.md
24
AGENTS.md
|
|
@ -219,6 +219,30 @@ logger::log_error("TAG", &format!("error: {}", e));
|
|||
- **Auth flow:** Email OTP + Yandex OAuth → deep link `talkis://auth?token=xxx`
|
||||
- **Subscription:** Free (own API key) or paid (cloud, 390₽/mo)
|
||||
|
||||
## Audio Pipeline Rules
|
||||
|
||||
For any audio, transcription, local STT, file transcription, or call-capture work, read
|
||||
`docs/audio-pipeline-principles.md` before editing code.
|
||||
|
||||
Key source files:
|
||||
|
||||
- Voice widget recording: `src/windows/widget/hooks/useWidgetRecording.ts`, `src/windows/widget/services/recordingRuntime.ts`
|
||||
- Native voice recording: `src-tauri/src/native_voice_recorder.rs`
|
||||
- STT request orchestration and hallucination filtering: `src-tauri/src/ai.rs`
|
||||
- Media conversion and chunking: `src-tauri/src/media.rs`
|
||||
- Managed local Whisper runtime: `src-tauri/src/bin/talkis-stt.rs`
|
||||
- Call capture and call transcript assembly: `src-tauri/src/call_capture.rs`, `src/lib/callCapture.ts`
|
||||
|
||||
Stable principles:
|
||||
|
||||
- Ordinary voice dictation should avoid ffmpeg in the hot path. Prefer native microphone capture that returns `audio/wav`, `16 kHz`, mono, PCM16.
|
||||
- Keep WebView `MediaRecorder` as a fallback, especially when a selected microphone can only be addressed reliably by WebView `deviceId`.
|
||||
- Local STT input must be `WAV 16 kHz mono PCM16`; skip ffmpeg when audio already matches that format.
|
||||
- Keep ffmpeg for arbitrary files, video, unsupported formats, diarization preparation, and file chunking.
|
||||
- Long local Whisper jobs can hallucinate repeated caption-like text on silence. Preserve the no-context local runtime settings and the repetitive-text filters unless a replacement is tested against long silent recordings.
|
||||
- macOS call system-audio capture is the only implemented system-audio path. Windows/Linux call capture should remain explicit unsupported placeholders until WASAPI loopback / PipeWire monitor capture is implemented.
|
||||
- Every audio path needs structured logs with enough evidence to debug runtime behavior: recorder stats, ffmpeg timing, STT endpoint, chunk index/size, and call-capture levels.
|
||||
|
||||
## Release Workflow
|
||||
|
||||
- Follow `docs/release/rule.md` for every release
|
||||
|
|
|
|||
18
README.md
18
README.md
|
|
@ -11,6 +11,7 @@ It sits in a small floating widget, listens while you hold a hotkey, sends audio
|
|||
- The recognized text is pasted automatically into the current app
|
||||
- A second press during recording locks the recording mode
|
||||
- The floating widget can start/stop recording with a mouse click, copy the latest result, and show a low microphone signal notice
|
||||
- Voice dictation uses native microphone capture first and produces `16 kHz` mono PCM WAV for local STT, with WebView `MediaRecorder` kept as fallback for selected-device parity
|
||||
- Autostart can be enabled from settings
|
||||
- The settings window lets you choose language, microphone, model source, API adapter, text cleanup style, and transcribe audio/video files
|
||||
- The settings window supports system, light, and dark appearance modes; system mode follows macOS
|
||||
|
|
@ -165,7 +166,7 @@ Talkis checks for a new app version in the background after startup and then per
|
|||
|
||||
The `Файлы` tab can transcribe audio or video files without LLM cleanup. File selection and drag-and-drop use a native path-based pipeline, so large files do not need to be loaded into the webview memory.
|
||||
|
||||
Talkis supports file transcription up to 1 GB. Video files, long recordings, and less common formats are converted inside the app with the bundled ffmpeg sidecar, split into safe audio chunks, and transcribed sequentially. The UI shows chunk progress while processing.
|
||||
Talkis supports file transcription up to 8 GB. Video files, long recordings, and less common formats are converted inside the app with the bundled ffmpeg sidecar, split into safe audio chunks, and transcribed sequentially. The UI shows chunk progress while processing. Ready `16 kHz` mono PCM WAV files can skip ffmpeg and go directly to local STT when they fit in one request.
|
||||
|
||||
File transcription can optionally split the transcript by speakers. In Talkis Cloud mode, `Разделить по говорящим` sends that file job to the cloud diarization endpoint on `proxy.talkis.ru`, backed by AssemblyAI, and does not use the installed local Whisper runtime. If cloud diarization is unavailable, Talkis stops with an error instead of silently falling back to local processing. In API or local mode, Talkis uses a downloaded local Whisper model with timestamps plus the speaker-diarization components for that file job. The global API or local model selection is not overwritten by this background diarization flow.
|
||||
|
||||
|
|
@ -242,6 +243,12 @@ If LLM model is set to "Без обработки", the raw transcription is pas
|
|||
- Delete and reinstall the model if the runtime reports missing model files
|
||||
- For custom localhost STT servers on non-managed ports, Talkis treats the endpoint as an external OpenAI-compatible server and does not try to start or stop it
|
||||
|
||||
### Local call/file transcription repeats the same phrase
|
||||
|
||||
- Long low-signal or silent regions can make local Whisper repeat caption-like phrases such as `Спасибо` or `Продолжение следует`
|
||||
- Talkis disables context carry-over in the managed Whisper runtime and filters repeated timestamped segments before speaker assembly
|
||||
- If this appears again, check `~/.talkis/talkis.log` for recorder stats, ffmpeg timings, STT chunk sizes, and call-capture levels
|
||||
|
||||
### The microphone list is empty
|
||||
|
||||
- Grant microphone permission in macOS
|
||||
|
|
@ -271,10 +278,10 @@ On Ubuntu/Debian, install the native Tauri and local sidecar build dependencies
|
|||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y libwebkit2gtk-4.1-dev libayatana-appindicator3-dev libxdo-dev librsvg2-dev patchelf clang libclang-dev cmake
|
||||
sudo apt install -y libwebkit2gtk-4.1-dev libayatana-appindicator3-dev libxdo-dev libasound2-dev librsvg2-dev patchelf clang libclang-dev cmake
|
||||
```
|
||||
|
||||
`libclang-dev` is required by `whisper-rs-sys` while building the local STT sidecars, `cmake` is required by the bundled `whisper.cpp` build, and `libxdo-dev` provides the `xdo` library used by Tauri/global hotkey linking. If libclang is installed in a non-standard location, set `LIBCLANG_PATH` to the directory that contains `libclang.so`.
|
||||
`libclang-dev` is required by `whisper-rs-sys` while building the local STT sidecars, `cmake` is required by the bundled `whisper.cpp` build, `libxdo-dev` provides the `xdo` library used by Tauri/global hotkey linking, and `libasound2-dev` is required by the native microphone recorder on Linux. If libclang is installed in a non-standard location, set `LIBCLANG_PATH` to the directory that contains `libclang.so`.
|
||||
|
||||
```bash
|
||||
bun install
|
||||
|
|
@ -303,7 +310,7 @@ The repository includes a GitHub Actions workflow at `.github/workflows/release.
|
|||
|
||||
- The canonical release process is documented in `docs/release/rule.md`
|
||||
- Before every release, refresh `README.md` and create a release review file from `docs/release/review-template.md`
|
||||
- Push a tag like `v0.1.19` to build and publish a GitHub Release
|
||||
- Push a tag like `v0.1.22` to build and publish a GitHub Release
|
||||
- Or run the workflow manually and provide a tag
|
||||
- The current workflow publishes macOS, Windows, and Linux release artifacts plus updater metadata
|
||||
- For macOS release builds, move `Talkis.app` to `Applications` before granting Accessibility access
|
||||
|
|
@ -324,6 +331,7 @@ Without Apple secrets, the workflow can still produce unsigned macOS release art
|
|||
- Tauri v2
|
||||
- React + TypeScript
|
||||
- Rust (backend, CGEvent paste, prompt engine)
|
||||
- Native microphone recording via `cpal`
|
||||
- OpenAI Whisper / gpt-4o-transcribe
|
||||
- OpenAI GPT-4o mini (text cleanup)
|
||||
- Talkis-managed local STT runtimes for Whisper, Qwen ASR, NVIDIA Parakeet MLX, and speaker diarization
|
||||
|
|
@ -331,4 +339,4 @@ Without Apple secrets, the workflow can still produce unsigned macOS release art
|
|||
|
||||
## Status
|
||||
|
||||
Talkis is an active work in progress. Current version: **0.1.21**.
|
||||
Talkis is an active work in progress. Current version: **0.1.22**.
|
||||
|
|
|
|||
236
docs/audio-pipeline-principles.md
Normal file
236
docs/audio-pipeline-principles.md
Normal file
|
|
@ -0,0 +1,236 @@
|
|||
# Audio Pipeline Principles
|
||||
|
||||
This document is the working contract for Talkis audio changes. Future agents should update it when the audio architecture changes.
|
||||
|
||||
## Goals
|
||||
|
||||
- Keep ordinary voice dictation fast.
|
||||
- Keep transcription quality stable.
|
||||
- Avoid moving expensive conversion into hot paths.
|
||||
- Keep local, cloud, file, and call behavior explicit instead of relying on hidden fallbacks.
|
||||
- Preserve enough logs to diagnose audio quality and runtime behavior from `~/.talkis/talkis.log`.
|
||||
|
||||
## Pipeline Map
|
||||
|
||||
Voice dictation:
|
||||
|
||||
```text
|
||||
Widget hotkey/button
|
||||
-> useWidgetRecording.ts
|
||||
-> recordingRuntime.ts
|
||||
-> native_voice_recorder.rs first
|
||||
-> processRecordingBlob()
|
||||
-> ai::transcribe_and_clean()
|
||||
-> paste_text
|
||||
```
|
||||
|
||||
File transcription:
|
||||
|
||||
```text
|
||||
Files tab / widget file drop
|
||||
-> src/lib/fileTranscription.ts
|
||||
-> ai::transcribe_file_path()
|
||||
-> media.rs preparation/chunking
|
||||
-> local, custom, or cloud STT
|
||||
```
|
||||
|
||||
Call transcription:
|
||||
|
||||
```text
|
||||
Widget call mode
|
||||
-> src/lib/callCapture.ts
|
||||
-> call_capture.rs for system track
|
||||
-> recordingRuntime.ts for mic track
|
||||
-> transcribeCallCaptureSession()
|
||||
-> file transcription pipeline
|
||||
```
|
||||
|
||||
## Voice Dictation
|
||||
|
||||
The primary voice path is native Rust capture:
|
||||
|
||||
- `src-tauri/src/native_voice_recorder.rs`
|
||||
- Tauri commands:
|
||||
- `start_native_voice_recording`
|
||||
- `pause_native_voice_recording`
|
||||
- `resume_native_voice_recording`
|
||||
- `stop_native_voice_recording`
|
||||
- Output contract:
|
||||
- `audio_base64`
|
||||
- `mime_type: "audio/wav"`
|
||||
- `file_name: "recording.wav"`
|
||||
- `duration_ms`
|
||||
- `sample_rate: 16000`
|
||||
- `channels: 1`
|
||||
- `peak`
|
||||
- `rms`
|
||||
|
||||
The recorder uses `cpal`, stores microphone samples in memory, converts to mono, resamples to `16 kHz`, writes PCM16 WAV, and logs stats on stop.
|
||||
|
||||
Important implementation detail: on macOS, `cpal::Stream` is not safe to keep in a global static. Keep the stream alive on its own recorder thread and store only thread-safe control handles in global state.
|
||||
|
||||
## Voice Fallback
|
||||
|
||||
Keep WebView `MediaRecorder` fallback in `recordingRuntime.ts`.
|
||||
|
||||
Fallback is required when:
|
||||
|
||||
- native recorder fails to start;
|
||||
- a selected microphone exists, but only WebView `deviceId` can identify it reliably;
|
||||
- platform-specific microphone permissions or device routing behave differently than `cpal`.
|
||||
|
||||
Do not remove the fallback unless selected-microphone parity is proven on macOS, Windows, and Linux.
|
||||
|
||||
## Local STT Input Format
|
||||
|
||||
Managed local Whisper expects:
|
||||
|
||||
```text
|
||||
WAV, 16 kHz, mono, PCM 16-bit
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Native voice recording should already produce this format.
|
||||
- `media::convert_audio_to_local_stt_wav()` must skip ffmpeg when input is already ready.
|
||||
- File transcription should also skip ffmpeg for ready WAV files that fit into one STT request.
|
||||
- ffmpeg remains the correct path for arbitrary audio/video, WebM/Opus, MP3/M4A/MP4, diarization prep, and chunking.
|
||||
|
||||
When editing conversion code, preserve the logs:
|
||||
|
||||
- `Running bundled ffmpeg sidecar`
|
||||
- `Bundled ffmpeg sidecar finished in ...ms`
|
||||
- `System ffmpeg fallback finished in ...ms`
|
||||
- `Skipping ffmpeg for local STT...`
|
||||
- `Skipping ffmpeg for file transcription...`
|
||||
|
||||
## Local Whisper Hallucination Guardrails
|
||||
|
||||
Long local Whisper jobs can produce repeated caption-like text on silence, for example:
|
||||
|
||||
- `Спасибо. Спасибо. Спасибо.`
|
||||
- `Продолжение следует...`
|
||||
- repeated copies of the last real phrase
|
||||
|
||||
This is not a UI recursion bug. It usually means Whisper received a long low-signal or silent region and reused context across internal windows.
|
||||
|
||||
Preserve these safeguards:
|
||||
|
||||
- `src-tauri/src/bin/talkis-stt.rs`
|
||||
- `params.set_no_context(true)`
|
||||
- `params.set_suppress_nst(true)`
|
||||
- low temperature / no temperature increment
|
||||
- entropy threshold
|
||||
- `src-tauri/src/ai.rs`
|
||||
- known hallucination detection
|
||||
- repetitive transcript text sanitizer
|
||||
- repetitive timestamped segment filter before diarization assembly
|
||||
|
||||
If changing these filters, test against:
|
||||
|
||||
- short real voice dictation;
|
||||
- long meeting/call recording with pauses;
|
||||
- mostly silent audio;
|
||||
- file transcription with speaker diarization enabled.
|
||||
|
||||
## File Transcription
|
||||
|
||||
File transcription is path-based. Do not load large files into WebView memory.
|
||||
|
||||
Rules:
|
||||
|
||||
- Keep `src/lib/fileTranscription.ts` as the frontend entry point.
|
||||
- Keep native path invocation through `transcribe_file_path`.
|
||||
- Keep chunk progress events and per-chunk logs.
|
||||
- Use ffmpeg for video and unsupported formats.
|
||||
- For local mode, each chunk must be converted to the local STT WAV contract before hitting the runtime.
|
||||
- For cloud mode, use the existing proxy endpoints and do not silently switch to local diarization.
|
||||
|
||||
Chunking currently protects API limits and long recordings. Do not remove chunking unless the target endpoint is proven to handle the full file size and duration.
|
||||
|
||||
## Call Capture
|
||||
|
||||
Call capture has two different tracks:
|
||||
|
||||
- mic track: user microphone, handled through the existing recording runtime / file pipeline;
|
||||
- system track: platform-specific system audio capture.
|
||||
|
||||
Current system-audio support:
|
||||
|
||||
- macOS: implemented via Core Audio process tap / aggregate device in `call_capture.rs`;
|
||||
- Windows: intentionally unsupported placeholder until WASAPI loopback is implemented;
|
||||
- Linux: intentionally unsupported placeholder until PipeWire monitor capture is implemented.
|
||||
|
||||
Do not present Windows/Linux system-call capture as working until platform capture is implemented and manually verified.
|
||||
|
||||
For macOS system track diagnostics, rely on stop-time logs:
|
||||
|
||||
```text
|
||||
System audio capture level: max=... dBFS, frames_above_noise_floor=...
|
||||
```
|
||||
|
||||
If `max=-120.0 dBFS` and `frames_above_noise_floor=0`, the system track is silent. The transcript should not treat that as usable remote-speaker audio.
|
||||
|
||||
## Speaker Diarization
|
||||
|
||||
Local speaker diarization uses:
|
||||
|
||||
- local Whisper segments with timestamps;
|
||||
- local diarization runtime segments;
|
||||
- overlap/nearest matching in `ai.rs`;
|
||||
- final formatting in `format_speaker_transcript()`.
|
||||
|
||||
Rules:
|
||||
|
||||
- Do not assemble speaker transcripts from STT text without timestamps.
|
||||
- Filter known repeated/hallucinated STT segments before assigning speakers.
|
||||
- If the system track has no diarizable speech, the existing mic fallback is acceptable, but logs must make that explicit.
|
||||
- Speaker labels shown to users should stay product-facing: `Вы`, `Гость N`.
|
||||
|
||||
## Logging Contract
|
||||
|
||||
Audio bugs are usually runtime bugs, not static type bugs. Keep logs specific.
|
||||
|
||||
Required evidence:
|
||||
|
||||
- recorder path used: native WAV, WebView WAV, WebM, or fallback;
|
||||
- selected mic and active device label when available;
|
||||
- audio stats: duration, sample rate, channels, peak, RMS;
|
||||
- ffmpeg start and finish timing;
|
||||
- STT endpoint and response status;
|
||||
- file chunk index, total chunks, and chunk size;
|
||||
- call system capture level and source/stored format.
|
||||
|
||||
Do not log API keys, device tokens, or full local model paths if they include sensitive user names.
|
||||
|
||||
## Verification
|
||||
|
||||
Minimum checks after audio pipeline edits:
|
||||
|
||||
```bash
|
||||
bunx tsc --noEmit
|
||||
cargo check --manifest-path src-tauri/Cargo.toml
|
||||
git diff --check
|
||||
```
|
||||
|
||||
For Rust logic in `ai.rs`, run targeted tests:
|
||||
|
||||
```bash
|
||||
cargo test --manifest-path src-tauri/Cargo.toml ai::tests --lib
|
||||
```
|
||||
|
||||
Manual checks when behavior changes:
|
||||
|
||||
- macOS voice dictation recognizes a real phrase and logs native recorder stats.
|
||||
- Local voice dictation does not run bundled ffmpeg when native WAV is used.
|
||||
- WebView fallback still records when native capture fails or selected mic cannot be mapped.
|
||||
- Ready `16 kHz mono PCM WAV` file skips ffmpeg.
|
||||
- MP3/MP4/WebM files still go through ffmpeg.
|
||||
- Long local call/file transcription with pauses does not produce repeated `Спасибо` / `Продолжение следует`.
|
||||
- macOS call capture does not regress; Windows/Linux call system capture remains clearly unsupported.
|
||||
|
||||
## Release Notes For Audio Dependencies
|
||||
|
||||
`cpal` pulls platform audio backends.
|
||||
|
||||
Linux release jobs need `libasound2-dev` for ALSA builds. If changing `cpal` features or replacing the recorder backend, re-check `.github/workflows/release.yml` and Linux Tauri dependency installation.
|
||||
79
docs/release/review-v0.1.22.md
Normal file
79
docs/release/review-v0.1.22.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
# Release Review v0.1.22
|
||||
|
||||
## Release
|
||||
|
||||
- Version: `0.1.22`
|
||||
- Release branch: `release/v0.1.22`
|
||||
- Target tag: `v0.1.22`
|
||||
- Reviewer: Codex
|
||||
- Date: 2026-05-22
|
||||
|
||||
## Scope
|
||||
|
||||
- Key changes included in this release:
|
||||
- Native Rust voice recorder using `cpal`, producing `16 kHz` mono PCM16 WAV for ordinary dictation.
|
||||
- WebView `MediaRecorder` retained as fallback when native capture cannot preserve selected-device behavior.
|
||||
- ffmpeg skip path for ready `16 kHz` mono PCM WAV in local STT and file transcription.
|
||||
- ffmpeg timing logs for bundled/system converter runs.
|
||||
- Local Whisper guardrails for long/silent audio: no context carry-over plus repetitive text/segment filtering.
|
||||
- Audio pipeline operating principles documented for future agents.
|
||||
- Stable release download aliases prepared in the GitHub release workflow.
|
||||
- Linux release dependencies updated with `libasound2-dev` for native recorder builds.
|
||||
- User-facing changes:
|
||||
- Faster local voice dictation path when native WAV capture is available.
|
||||
- Reduced chance of repeated `Спасибо` / `Продолжение следует` hallucinations in long local call/file transcripts.
|
||||
- Ready local-STT WAV files can avoid unnecessary conversion.
|
||||
- README now documents the current `0.1.22` behavior and audio troubleshooting.
|
||||
- Risky areas:
|
||||
- Cross-platform microphone capture through `cpal`.
|
||||
- Selected microphone parity between native device labels and WebView `deviceId`.
|
||||
- Long local STT recordings with large silent regions.
|
||||
- Release workflow on Linux due to the new ALSA build dependency.
|
||||
|
||||
## Checks run
|
||||
|
||||
- `bun run check:release`: passed.
|
||||
- `TAURI_SIGNING_PRIVATE_KEY_PATH=/Users/trixter/.tauri/talkis-updater.key bun run build:release:macos`: app, DMG, and updater tarball were built; command failed at updater signing with `incorrect updater private key password: Device not configured (os error 6)`.
|
||||
- Native/GitHub Windows build: not run locally; expected to run in GitHub Actions matrix.
|
||||
- Native/GitHub Linux build: not run locally; expected to run in GitHub Actions matrix with `libasound2-dev`.
|
||||
- Additional checks:
|
||||
- `bun run check:versions`: passed.
|
||||
- `cargo check --manifest-path src-tauri/Cargo.toml`: passed.
|
||||
- `cargo test --manifest-path src-tauri/Cargo.toml ai::tests --lib`: passed.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
## Manual review
|
||||
|
||||
- Hotkey flow:
|
||||
- FSM smoke tests passed through `bun run check:release`.
|
||||
- Recording flow now tries native capture first and falls back to WebView capture on native failure.
|
||||
- Onboarding permissions:
|
||||
- No onboarding UI or permission text changes in this release.
|
||||
- Widget position and notice behavior:
|
||||
- No widget positioning changes in this release.
|
||||
- Low microphone monitor remains on the WebView fallback path; native path relies on native recorder stats.
|
||||
- Transcription quality and short-utterance handling:
|
||||
- Existing short hallucination handling kept.
|
||||
- New repetitive text filters covered by targeted unit tests.
|
||||
- Manual evidence from logs showed native dictation recognized `Раз, два, три, четыре, пять`.
|
||||
- README refreshed:
|
||||
- Updated current version to `0.1.22`.
|
||||
- Added native voice capture, 8 GB file limit, ready-WAV fast path, local repetition troubleshooting, and Linux `libasound2-dev`.
|
||||
|
||||
## Findings
|
||||
|
||||
- Blockers:
|
||||
- None for pushing the GitHub Actions release, assuming repository updater signing secrets remain valid as in prior releases.
|
||||
- Non-blocking issues:
|
||||
- Local macOS release signing could not be validated because the updater private key password is not configured in this local shell.
|
||||
- Full `cargo test --lib` is not used as a release gate here because an unrelated existing `prompt_config::tests::temperature_is_set_per_style` expectation is stale (`tech` config returns `0.1`, test expects `0.15`).
|
||||
- Windows/Linux native recorder behavior is not manually verified on physical machines in this local release prep.
|
||||
- Follow-ups after release:
|
||||
- Watch the GitHub Actions release matrix for Linux ALSA/cpal build issues.
|
||||
- Manually test Windows/Linux dictation on released artifacts.
|
||||
- Fix the stale prompt temperature unit test in a separate cleanup.
|
||||
|
||||
## Decision
|
||||
|
||||
- Ready for `main` merge: yes.
|
||||
- Ready for tag publish: yes, with signing delegated to GitHub Actions secrets.
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"name": "talkis",
|
||||
"private": true,
|
||||
"version": "0.1.21",
|
||||
"version": "0.1.22",
|
||||
"type": "module",
|
||||
"packageManager": "bun@1.2.13",
|
||||
"scripts": {
|
||||
|
|
|
|||
182
src-tauri/Cargo.lock
generated
182
src-tauri/Cargo.lock
generated
|
|
@ -32,6 +32,28 @@ dependencies = [
|
|||
"alloc-no-stdlib",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "alsa"
|
||||
version = "0.9.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ed7572b7ba83a31e20d1b48970ee402d2e3e0537dcfe0a3ff4d6eb7508617d43"
|
||||
dependencies = [
|
||||
"alsa-sys",
|
||||
"bitflags 2.11.0",
|
||||
"cfg-if",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "alsa-sys"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "db8fee663d06c4e303404ef5f40488a53e062f89ba8bfed81f42325aafad1527"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"pkg-config",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "android_system_properties"
|
||||
version = "0.1.5"
|
||||
|
|
@ -494,6 +516,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||
checksum = "aebf35691d1bfb0ac386a69bac2fde4dd276fb618cf8bf4f5318fe285e821bb2"
|
||||
dependencies = [
|
||||
"find-msvc-tools",
|
||||
"jobserver",
|
||||
"libc",
|
||||
"shlex",
|
||||
]
|
||||
|
||||
|
|
@ -711,6 +735,49 @@ dependencies = [
|
|||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "coreaudio-rs"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "321077172d79c662f64f5071a03120748d5bb652f5231570141be24cfcd2bace"
|
||||
dependencies = [
|
||||
"bitflags 1.3.2",
|
||||
"core-foundation-sys",
|
||||
"coreaudio-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "coreaudio-sys"
|
||||
version = "0.2.17"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ceec7a6067e62d6f931a2baf6f3a751f4a892595bcec1461a3c94ef9949864b6"
|
||||
dependencies = [
|
||||
"bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cpal"
|
||||
version = "0.15.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "873dab07c8f743075e57f524c583985fbaf745602acbe916a01539364369a779"
|
||||
dependencies = [
|
||||
"alsa",
|
||||
"core-foundation-sys",
|
||||
"coreaudio-rs",
|
||||
"dasp_sample",
|
||||
"jni",
|
||||
"js-sys",
|
||||
"libc",
|
||||
"mach2",
|
||||
"ndk 0.8.0",
|
||||
"ndk-context",
|
||||
"oboe",
|
||||
"wasm-bindgen",
|
||||
"wasm-bindgen-futures",
|
||||
"web-sys",
|
||||
"windows 0.54.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.2.17"
|
||||
|
|
@ -845,6 +912,12 @@ dependencies = [
|
|||
"syn 2.0.117",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dasp_sample"
|
||||
version = "0.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0c87e182de0887fd5361989c677c4e8f5000cd9491d6d563161a8f3a5519fc7f"
|
||||
|
||||
[[package]]
|
||||
name = "deranged"
|
||||
version = "0.5.8"
|
||||
|
|
@ -2351,6 +2424,16 @@ version = "0.3.0"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8eaf4bc02d17cbdd7ff4c7438cafcdf7fb9a4613313ad11b4f8fefe7d3fa0130"
|
||||
|
||||
[[package]]
|
||||
name = "jobserver"
|
||||
version = "0.1.34"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33"
|
||||
dependencies = [
|
||||
"getrandom 0.3.4",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "js-sys"
|
||||
version = "0.3.91"
|
||||
|
|
@ -2507,6 +2590,15 @@ version = "0.1.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c41e0c4fef86961ac6d6f8a82609f55f31b05e4fce149ac5710e439df7619ba4"
|
||||
|
||||
[[package]]
|
||||
name = "mach2"
|
||||
version = "0.4.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d640282b302c0bb0a2a8e0233ead9035e3bed871f0b7e81fe4a1ec829765db44"
|
||||
dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "markup5ever"
|
||||
version = "0.14.1"
|
||||
|
|
@ -2670,6 +2762,20 @@ dependencies = [
|
|||
"tempfile",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ndk"
|
||||
version = "0.8.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2076a31b7010b17a38c01907c45b945e8f11495ee4dd588309718901b1f7a5b7"
|
||||
dependencies = [
|
||||
"bitflags 2.11.0",
|
||||
"jni-sys",
|
||||
"log",
|
||||
"ndk-sys 0.5.0+25.2.9519653",
|
||||
"num_enum",
|
||||
"thiserror 1.0.69",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ndk"
|
||||
version = "0.9.0"
|
||||
|
|
@ -2679,7 +2785,7 @@ dependencies = [
|
|||
"bitflags 2.11.0",
|
||||
"jni-sys",
|
||||
"log",
|
||||
"ndk-sys",
|
||||
"ndk-sys 0.6.0+11769913",
|
||||
"num_enum",
|
||||
"raw-window-handle",
|
||||
"thiserror 1.0.69",
|
||||
|
|
@ -2691,6 +2797,15 @@ version = "0.1.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "27b02d87554356db9e9a873add8782d4ea6e3e58ea071a9adb9a2e8ddb884a8b"
|
||||
|
||||
[[package]]
|
||||
name = "ndk-sys"
|
||||
version = "0.5.0+25.2.9519653"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8c196769dd60fd4f363e11d948139556a344e79d451aeb2fa2fd040738ef7691"
|
||||
dependencies = [
|
||||
"jni-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ndk-sys"
|
||||
version = "0.6.0+11769913"
|
||||
|
|
@ -2737,6 +2852,17 @@ version = "0.2.0"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050"
|
||||
|
||||
[[package]]
|
||||
name = "num-derive"
|
||||
version = "0.4.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ed3955f1a9c7c0c15e092f9c887db08b1fc683305fdf6eb6684f22555355e202"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.117",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-traits"
|
||||
version = "0.2.19"
|
||||
|
|
@ -3010,6 +3136,29 @@ dependencies = [
|
|||
"objc2-foundation",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "oboe"
|
||||
version = "0.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e8b61bebd49e5d43f5f8cc7ee2891c16e0f41ec7954d36bcb6c14c5e0de867fb"
|
||||
dependencies = [
|
||||
"jni",
|
||||
"ndk 0.8.0",
|
||||
"ndk-context",
|
||||
"num-derive",
|
||||
"num-traits",
|
||||
"oboe-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "oboe-sys"
|
||||
version = "0.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6c8bb09a4a2b1d668170cfe0a7d5bc103f8999fb316c98099b6a9939c9f2e79d"
|
||||
dependencies = [
|
||||
"cc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "once_cell"
|
||||
version = "1.21.3"
|
||||
|
|
@ -4483,7 +4632,7 @@ checksum = "aac18da81ebbf05109ab275b157c22a653bb3c12cf884450179942f81bcbf6c3"
|
|||
dependencies = [
|
||||
"bytemuck",
|
||||
"js-sys",
|
||||
"ndk",
|
||||
"ndk 0.9.0",
|
||||
"objc2 0.6.4",
|
||||
"objc2-core-foundation",
|
||||
"objc2-core-graphics",
|
||||
|
|
@ -4679,11 +4828,12 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "talkis"
|
||||
version = "0.1.21"
|
||||
version = "0.1.22"
|
||||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"block2 0.6.2",
|
||||
"chrono",
|
||||
"cpal",
|
||||
"dirs 5.0.1",
|
||||
"enigo",
|
||||
"hex",
|
||||
|
|
@ -4739,9 +4889,9 @@ dependencies = [
|
|||
"jni",
|
||||
"libc",
|
||||
"log",
|
||||
"ndk",
|
||||
"ndk 0.9.0",
|
||||
"ndk-context",
|
||||
"ndk-sys",
|
||||
"ndk-sys 0.6.0+11769913",
|
||||
"objc2 0.6.4",
|
||||
"objc2-app-kit",
|
||||
"objc2-foundation",
|
||||
|
|
@ -6256,6 +6406,16 @@ dependencies = [
|
|||
"windows-version",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows"
|
||||
version = "0.54.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9252e5725dbed82865af151df558e754e4a3c2c30818359eb17465f1346a1b49"
|
||||
dependencies = [
|
||||
"windows-core 0.54.0",
|
||||
"windows-targets 0.52.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows"
|
||||
version = "0.56.0"
|
||||
|
|
@ -6288,6 +6448,16 @@ dependencies = [
|
|||
"windows-core 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-core"
|
||||
version = "0.54.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "12661b9c89351d684a50a8a643ce5f608e20243b9fb84687800163429f161d65"
|
||||
dependencies = [
|
||||
"windows-result 0.1.2",
|
||||
"windows-targets 0.52.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-core"
|
||||
version = "0.56.0"
|
||||
|
|
@ -6954,7 +7124,7 @@ dependencies = [
|
|||
"javascriptcore-rs",
|
||||
"jni",
|
||||
"libc",
|
||||
"ndk",
|
||||
"ndk 0.9.0",
|
||||
"objc2 0.6.4",
|
||||
"objc2-app-kit",
|
||||
"objc2-core-foundation",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
[package]
|
||||
name = "talkis"
|
||||
version = "0.1.21"
|
||||
version = "0.1.22"
|
||||
description = "Talkis - Voice to Text Desktop Widget"
|
||||
authors = ["trixter"]
|
||||
edition = "2021"
|
||||
|
|
@ -28,6 +28,7 @@ sha2 = "0.10"
|
|||
hex = "0.4"
|
||||
base64 = "0.22"
|
||||
chrono = { version = "0.4", features = ["serde"] }
|
||||
cpal = "0.15.3"
|
||||
window-vibrancy = "0.7.1"
|
||||
dirs = "5"
|
||||
include_dir = "0.7"
|
||||
|
|
|
|||
|
|
@ -237,6 +237,156 @@ fn is_likely_short_uncertain_transcription(
|
|||
all_high_no_speech || all_low_confidence || any_high_compression
|
||||
}
|
||||
|
||||
fn normalize_repetition_key(value: &str) -> String {
|
||||
let normalized = value.trim().to_lowercase().replace('ё', "е");
|
||||
let mut key = String::with_capacity(normalized.len());
|
||||
|
||||
for ch in normalized.chars() {
|
||||
if ch.is_alphanumeric() {
|
||||
key.push(ch);
|
||||
} else {
|
||||
key.push(' ');
|
||||
}
|
||||
}
|
||||
|
||||
key.split_whitespace().collect::<Vec<_>>().join(" ")
|
||||
}
|
||||
|
||||
fn is_known_caption_artifact_key(key: &str) -> bool {
|
||||
key == "продолжение следует"
|
||||
|| key == "to be continued"
|
||||
|| key == "thanks for watching"
|
||||
|| key == "спасибо за просмотр"
|
||||
}
|
||||
|
||||
fn split_transcript_units(text: &str) -> Vec<String> {
|
||||
let mut units = Vec::new();
|
||||
let mut current = String::new();
|
||||
|
||||
for ch in text.chars() {
|
||||
current.push(ch);
|
||||
if matches!(ch, '.' | '!' | '?' | '…' | '\n') {
|
||||
let trimmed = current.trim();
|
||||
if !normalize_repetition_key(trimmed).is_empty() {
|
||||
units.push(trimmed.to_string());
|
||||
}
|
||||
current.clear();
|
||||
}
|
||||
}
|
||||
|
||||
let trimmed = current.trim();
|
||||
if !normalize_repetition_key(trimmed).is_empty() {
|
||||
units.push(trimmed.to_string());
|
||||
}
|
||||
|
||||
if units.is_empty() && !text.trim().is_empty() {
|
||||
units.push(text.trim().to_string());
|
||||
}
|
||||
|
||||
units
|
||||
}
|
||||
|
||||
fn sanitize_repetitive_transcript_text(text: &str) -> String {
|
||||
let units = split_transcript_units(text);
|
||||
if units.is_empty() {
|
||||
return String::new();
|
||||
}
|
||||
|
||||
let mut totals = std::collections::HashMap::<String, usize>::new();
|
||||
for unit in &units {
|
||||
let key = normalize_repetition_key(unit);
|
||||
if !key.is_empty() {
|
||||
*totals.entry(key).or_insert(0) += 1;
|
||||
}
|
||||
}
|
||||
|
||||
let mut kept_counts = std::collections::HashMap::<String, usize>::new();
|
||||
let mut kept = Vec::new();
|
||||
|
||||
for unit in units {
|
||||
let key = normalize_repetition_key(&unit);
|
||||
if key.is_empty() || is_known_caption_artifact_key(&key) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let total = totals.get(&key).copied().unwrap_or(0);
|
||||
if total >= 3 {
|
||||
let count = kept_counts.entry(key).or_insert(0);
|
||||
if *count >= 1 {
|
||||
continue;
|
||||
}
|
||||
*count += 1;
|
||||
}
|
||||
|
||||
kept.push(unit);
|
||||
}
|
||||
|
||||
kept.join(" ").trim().to_string()
|
||||
}
|
||||
|
||||
fn is_repeated_segment_filter_candidate(key: &str) -> bool {
|
||||
if is_known_caption_artifact_key(key) {
|
||||
return true;
|
||||
}
|
||||
|
||||
let token_count = key.split_whitespace().count();
|
||||
key == "спасибо" || (2..=12).contains(&token_count)
|
||||
}
|
||||
|
||||
fn sanitize_stt_segments(segments: Vec<SttTranscriptSegment>) -> Vec<SttTranscriptSegment> {
|
||||
let mut cleaned = Vec::with_capacity(segments.len());
|
||||
for mut segment in segments {
|
||||
segment.text = sanitize_repetitive_transcript_text(&segment.text);
|
||||
if !segment.text.trim().is_empty() {
|
||||
cleaned.push(segment);
|
||||
}
|
||||
}
|
||||
|
||||
let mut totals = std::collections::HashMap::<String, usize>::new();
|
||||
for segment in &cleaned {
|
||||
let key = normalize_repetition_key(&segment.text);
|
||||
if !key.is_empty() {
|
||||
*totals.entry(key).or_insert(0) += 1;
|
||||
}
|
||||
}
|
||||
|
||||
let original_len = cleaned.len();
|
||||
let mut seen = std::collections::HashMap::<String, usize>::new();
|
||||
let filtered = cleaned
|
||||
.into_iter()
|
||||
.filter(|segment| {
|
||||
let key = normalize_repetition_key(&segment.text);
|
||||
if key.is_empty() || is_known_caption_artifact_key(&key) {
|
||||
return false;
|
||||
}
|
||||
|
||||
let total = totals.get(&key).copied().unwrap_or(0);
|
||||
if total >= 3 && is_repeated_segment_filter_candidate(&key) {
|
||||
let count = seen.entry(key).or_insert(0);
|
||||
if *count >= 1 {
|
||||
return false;
|
||||
}
|
||||
*count += 1;
|
||||
}
|
||||
|
||||
true
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
if filtered.len() != original_len {
|
||||
logger::log_info(
|
||||
"WHISPER",
|
||||
&format!(
|
||||
"Filtered repetitive local STT segments: before={}, after={}",
|
||||
original_len,
|
||||
filtered.len()
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
filtered
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
pub async fn transcribe_and_clean(
|
||||
app: AppHandle,
|
||||
|
|
@ -489,6 +639,33 @@ async fn transcribe_audio_bytes_internal(
|
|||
});
|
||||
}
|
||||
|
||||
let sanitized_raw = sanitize_repetitive_transcript_text(&raw);
|
||||
if sanitized_raw != raw {
|
||||
logger::log_info(
|
||||
"WHISPER",
|
||||
&format!(
|
||||
"Sanitized repetitive local STT text: before_chars={}, after_chars={}",
|
||||
raw.chars().count(),
|
||||
sanitized_raw.chars().count()
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
if sanitized_raw.is_empty() {
|
||||
logger::log_info(
|
||||
"WHISPER",
|
||||
&format!(
|
||||
"Detected likely repetitive silence hallucination, dropping transcription: \"{}\"",
|
||||
raw
|
||||
),
|
||||
);
|
||||
return Ok(AudioTranscriptionResult {
|
||||
raw: String::new(),
|
||||
cleaned: String::new(),
|
||||
segments: Vec::new(),
|
||||
});
|
||||
}
|
||||
|
||||
if is_known_whisper_hallucination(&raw) {
|
||||
logger::log_info(
|
||||
"WHISPER",
|
||||
|
|
@ -532,7 +709,7 @@ async fn transcribe_audio_bytes_internal(
|
|||
);
|
||||
logger::log_info("API", "Transcription complete");
|
||||
|
||||
let segments = whisper_body
|
||||
let raw_segments = whisper_body
|
||||
.segments
|
||||
.iter()
|
||||
.filter_map(|segment| {
|
||||
|
|
@ -554,16 +731,41 @@ async fn transcribe_audio_bytes_internal(
|
|||
})
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
let segments = sanitize_stt_segments(raw_segments);
|
||||
|
||||
if require_segments && segments.is_empty() {
|
||||
return Err(
|
||||
"Для разделения по говорящим нужна локальная Whisper-модель с таймкодами.".to_string(),
|
||||
if whisper_body.segments.is_empty() {
|
||||
return Err(
|
||||
"Для разделения по говорящим нужна локальная Whisper-модель с таймкодами."
|
||||
.to_string(),
|
||||
);
|
||||
}
|
||||
|
||||
logger::log_info(
|
||||
"WHISPER",
|
||||
"All timestamped local STT segments were filtered as repetitive hallucinations",
|
||||
);
|
||||
return Ok(AudioTranscriptionResult {
|
||||
raw: String::new(),
|
||||
cleaned: String::new(),
|
||||
segments: Vec::new(),
|
||||
});
|
||||
}
|
||||
|
||||
let cleaned_raw = if require_segments && !segments.is_empty() {
|
||||
segments
|
||||
.iter()
|
||||
.map(|segment| segment.text.trim())
|
||||
.filter(|text| !text.is_empty())
|
||||
.collect::<Vec<_>>()
|
||||
.join(" ")
|
||||
} else {
|
||||
sanitized_raw
|
||||
};
|
||||
|
||||
Ok(AudioTranscriptionResult {
|
||||
raw: raw.clone(),
|
||||
cleaned: raw,
|
||||
raw: cleaned_raw.clone(),
|
||||
cleaned: cleaned_raw,
|
||||
segments,
|
||||
})
|
||||
}
|
||||
|
|
@ -2142,6 +2344,41 @@ mod tests {
|
|||
assert_eq!(segments[0].text, "Давайте начнем");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn removes_caption_artifact_units_from_local_stt_text() {
|
||||
let text = sanitize_repetitive_transcript_text(
|
||||
"Продолжение следует... Продолжение следует... Сколько уйдет времени?",
|
||||
);
|
||||
|
||||
assert_eq!(text, "Сколько уйдет времени?");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compacts_repeated_sentence_runs_inside_segment() {
|
||||
let text = sanitize_repetitive_transcript_text(
|
||||
"Проверим пайплайн. Убедимся в том, что у нас все было. Убедимся в том, что у нас все было. Убедимся в том, что у нас все было.",
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
text,
|
||||
"Проверим пайплайн. Убедимся в том, что у нас все было."
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn filters_repeated_low_information_stt_segments() {
|
||||
let segments = sanitize_stt_segments(vec![
|
||||
stt(0.0, 1.0, "Спасибо."),
|
||||
stt(30.0, 31.0, "Спасибо."),
|
||||
stt(60.0, 61.0, "Спасибо."),
|
||||
stt(90.0, 94.0, "Реальная фраза"),
|
||||
]);
|
||||
|
||||
assert_eq!(segments.len(), 2);
|
||||
assert_eq!(segments[0].text, "Спасибо.");
|
||||
assert_eq!(segments[1].text, "Реальная фраза");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plain_response_keeps_legacy_raw_cleaned_shape() {
|
||||
let response = plain_transcribe_response("raw".to_string(), "cleaned".to_string());
|
||||
|
|
|
|||
|
|
@ -526,6 +526,11 @@ fn transcribe(request: &HttpRequest, config: &RuntimeConfig) -> Result<String, (
|
|||
params.set_print_progress(false);
|
||||
params.set_print_realtime(false);
|
||||
params.set_print_timestamps(false);
|
||||
params.set_no_context(true);
|
||||
params.set_suppress_nst(true);
|
||||
params.set_temperature(0.0);
|
||||
params.set_temperature_inc(0.0);
|
||||
params.set_entropy_thold(2.2);
|
||||
|
||||
if let Some(language) = multipart.fields.get("language") {
|
||||
let language = language.trim();
|
||||
|
|
|
|||
|
|
@ -4,8 +4,9 @@ mod commands;
|
|||
mod hotkey_capture;
|
||||
mod local_stt;
|
||||
mod logger;
|
||||
mod media_permissions;
|
||||
mod media;
|
||||
mod media_permissions;
|
||||
mod native_voice_recorder;
|
||||
mod paste;
|
||||
mod prompt_config;
|
||||
|
||||
|
|
@ -120,6 +121,10 @@ pub fn run() {
|
|||
call_capture::get_call_capture_status,
|
||||
call_capture::get_call_capture_duration_ms,
|
||||
media::prepare_media_for_transcription,
|
||||
native_voice_recorder::start_native_voice_recording,
|
||||
native_voice_recorder::pause_native_voice_recording,
|
||||
native_voice_recorder::resume_native_voice_recording,
|
||||
native_voice_recorder::stop_native_voice_recording,
|
||||
ai::test_api_connection,
|
||||
ai::list_stt_models,
|
||||
ai::install_stt_model,
|
||||
|
|
|
|||
|
|
@ -1,12 +1,14 @@
|
|||
use crate::logger;
|
||||
use base64::Engine;
|
||||
use hound::{SampleFormat, WavReader};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::env;
|
||||
use std::fs;
|
||||
use std::io::Cursor;
|
||||
use std::path::{Path, PathBuf};
|
||||
#[cfg(debug_assertions)]
|
||||
use std::process::Command;
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
use std::time::{Instant, SystemTime, UNIX_EPOCH};
|
||||
use tauri_plugin_shell::ShellExt;
|
||||
|
||||
const MAX_TRANSCRIPTION_BYTES: u64 = 25 * 1024 * 1024;
|
||||
|
|
@ -75,6 +77,18 @@ fn unique_temp_path(prefix: &str, extension: &str) -> PathBuf {
|
|||
))
|
||||
}
|
||||
|
||||
fn is_local_stt_ready_wav(input_bytes: &[u8]) -> bool {
|
||||
let Ok(reader) = WavReader::new(Cursor::new(input_bytes)) else {
|
||||
return false;
|
||||
};
|
||||
let spec = reader.spec();
|
||||
|
||||
spec.sample_rate == 16000
|
||||
&& spec.channels == 1
|
||||
&& spec.bits_per_sample == 16
|
||||
&& spec.sample_format == SampleFormat::Int
|
||||
}
|
||||
|
||||
#[cfg(debug_assertions)]
|
||||
fn ffmpeg_candidates() -> Vec<PathBuf> {
|
||||
let mut candidates = Vec::new();
|
||||
|
|
@ -112,16 +126,26 @@ async fn run_ffmpeg(app: &tauri::AppHandle, args: Vec<String>) -> Result<Vec<u8>
|
|||
match app.shell().sidecar("talkis-ffmpeg") {
|
||||
Ok(command) => {
|
||||
logger::log_info("MEDIA", "Running bundled ffmpeg sidecar");
|
||||
let started_at = Instant::now();
|
||||
let output = command
|
||||
.args(args.clone())
|
||||
.output()
|
||||
.await
|
||||
.map_err(|err| format!("Не удалось запустить встроенный ffmpeg: {}", err))?;
|
||||
let elapsed_ms = started_at.elapsed().as_millis();
|
||||
|
||||
if output.status.success() {
|
||||
logger::log_info(
|
||||
"MEDIA",
|
||||
&format!("Bundled ffmpeg sidecar finished in {}ms", elapsed_ms),
|
||||
);
|
||||
return Ok(output.stderr);
|
||||
}
|
||||
|
||||
logger::log_error(
|
||||
"MEDIA",
|
||||
&format!("Bundled ffmpeg sidecar failed in {}ms", elapsed_ms),
|
||||
);
|
||||
return Err(String::from_utf8_lossy(&output.stderr).trim().to_string());
|
||||
}
|
||||
Err(err) => {
|
||||
|
|
@ -139,15 +163,25 @@ async fn run_ffmpeg(app: &tauri::AppHandle, args: Vec<String>) -> Result<Vec<u8>
|
|||
"MEDIA",
|
||||
&format!("Running system ffmpeg fallback: {:?}", ffmpeg),
|
||||
);
|
||||
let started_at = Instant::now();
|
||||
let output = Command::new(&ffmpeg)
|
||||
.args(&args)
|
||||
.output()
|
||||
.map_err(|err| format!("Не удалось запустить системный ffmpeg: {}", err))?;
|
||||
let elapsed_ms = started_at.elapsed().as_millis();
|
||||
|
||||
if output.status.success() {
|
||||
logger::log_info(
|
||||
"MEDIA",
|
||||
&format!("System ffmpeg fallback finished in {}ms", elapsed_ms),
|
||||
);
|
||||
return Ok(output.stderr);
|
||||
}
|
||||
|
||||
logger::log_error(
|
||||
"MEDIA",
|
||||
&format!("System ffmpeg fallback failed in {}ms", elapsed_ms),
|
||||
);
|
||||
return Err(String::from_utf8_lossy(&output.stderr).trim().to_string());
|
||||
}
|
||||
|
||||
|
|
@ -162,6 +196,17 @@ pub async fn convert_audio_to_local_stt_wav(
|
|||
input_bytes: &[u8],
|
||||
file_name: &str,
|
||||
) -> Result<Vec<u8>, String> {
|
||||
if is_local_stt_ready_wav(input_bytes) {
|
||||
logger::log_info(
|
||||
"MEDIA",
|
||||
&format!(
|
||||
"Skipping ffmpeg for local STT: input is already 16 kHz mono PCM WAV, size={} bytes",
|
||||
input_bytes.len()
|
||||
),
|
||||
);
|
||||
return Ok(input_bytes.to_vec());
|
||||
}
|
||||
|
||||
let input_ext = file_extension(file_name);
|
||||
let input_path = unique_temp_path("local-stt-input", input_ext);
|
||||
let output_path = unique_temp_path("local-stt-output", "wav");
|
||||
|
|
@ -231,6 +276,34 @@ pub async fn prepare_media_file_chunks_for_transcription(
|
|||
);
|
||||
}
|
||||
|
||||
if metadata.len() <= MAX_TRANSCRIPTION_BYTES {
|
||||
if let Ok(input_bytes) = fs::read(input_path) {
|
||||
if is_local_stt_ready_wav(&input_bytes) {
|
||||
logger::log_info(
|
||||
"MEDIA",
|
||||
&format!(
|
||||
"Skipping ffmpeg for file transcription: input is already 16 kHz mono PCM WAV, size={} bytes",
|
||||
metadata.len()
|
||||
),
|
||||
);
|
||||
return Ok(PreparedMediaChunks {
|
||||
temp_dir: unique_temp_path("file-transcription-direct-wav", "dir"),
|
||||
chunks: vec![PreparedMediaChunk {
|
||||
path: input_path.to_path_buf(),
|
||||
file_name: input_path
|
||||
.file_name()
|
||||
.and_then(|value| value.to_str())
|
||||
.unwrap_or("talkis-transcription.wav")
|
||||
.to_string(),
|
||||
mime_type: "audio/wav".to_string(),
|
||||
size_bytes: metadata.len(),
|
||||
start_offset_seconds: 0.0,
|
||||
}],
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let chunks_dir = unique_temp_path("file-transcription-chunks", "dir");
|
||||
fs::create_dir_all(&chunks_dir)
|
||||
.map_err(|err| format!("Не удалось подготовить временную папку: {}", err))?;
|
||||
|
|
|
|||
562
src-tauri/src/native_voice_recorder.rs
Normal file
562
src-tauri/src/native_voice_recorder.rs
Normal file
|
|
@ -0,0 +1,562 @@
|
|||
use crate::logger;
|
||||
use base64::Engine;
|
||||
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::env;
|
||||
use std::fs;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::{mpsc, Arc, Mutex, OnceLock};
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
|
||||
const TARGET_SAMPLE_RATE: u32 = 16_000;
|
||||
const TARGET_CHANNELS: u16 = 1;
|
||||
const PCM_TARGET_PEAK: f32 = 0.82;
|
||||
const PCM_NORMALIZE_BELOW_PEAK: f32 = 0.35;
|
||||
const PCM_MIN_SIGNAL_PEAK: f32 = 0.001;
|
||||
const PCM_MAX_GAIN: f32 = 8.0;
|
||||
|
||||
static RECORDER: OnceLock<Mutex<Option<NativeVoiceRecorder>>> = OnceLock::new();
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct StartNativeVoiceRecordingRequest {
|
||||
pub device_label: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct NativeVoiceRecordingResult {
|
||||
pub audio_base64: String,
|
||||
pub mime_type: String,
|
||||
pub file_name: String,
|
||||
pub duration_ms: u64,
|
||||
pub sample_rate: u32,
|
||||
pub channels: u16,
|
||||
pub peak: f32,
|
||||
pub rms: f32,
|
||||
}
|
||||
|
||||
struct NativeVoiceRecorder {
|
||||
state: Arc<Mutex<NativeRecorderState>>,
|
||||
stop_tx: mpsc::Sender<()>,
|
||||
stopped_rx: mpsc::Receiver<Result<(), String>>,
|
||||
source_sample_rate: u32,
|
||||
source_channels: u16,
|
||||
device_name: String,
|
||||
}
|
||||
|
||||
struct NativeRecorderThreadInfo {
|
||||
source_sample_rate: u32,
|
||||
source_channels: u16,
|
||||
device_name: String,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct NativeRecorderState {
|
||||
samples: Vec<f32>,
|
||||
paused: bool,
|
||||
}
|
||||
|
||||
struct PcmStats {
|
||||
peak: f32,
|
||||
rms: f32,
|
||||
}
|
||||
|
||||
fn recorder_slot() -> &'static Mutex<Option<NativeVoiceRecorder>> {
|
||||
RECORDER.get_or_init(|| Mutex::new(None))
|
||||
}
|
||||
|
||||
fn unique_temp_path(prefix: &str, extension: &str) -> PathBuf {
|
||||
let now = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|duration| duration.as_nanos())
|
||||
.unwrap_or_default();
|
||||
let pid = std::process::id();
|
||||
env::temp_dir().join(format!(
|
||||
"talkis-{}-{}-{}.{}",
|
||||
prefix,
|
||||
pid,
|
||||
now,
|
||||
extension.trim_start_matches('.')
|
||||
))
|
||||
}
|
||||
|
||||
fn select_input_device(
|
||||
host: &cpal::Host,
|
||||
device_label: Option<&str>,
|
||||
) -> Result<cpal::Device, String> {
|
||||
if let Some(label) = device_label
|
||||
.map(str::trim)
|
||||
.filter(|value| !value.is_empty())
|
||||
{
|
||||
let label_lower = label.to_lowercase();
|
||||
match host.input_devices() {
|
||||
Ok(devices) => {
|
||||
for device in devices {
|
||||
let Ok(name) = device.name() else {
|
||||
continue;
|
||||
};
|
||||
if name == label || name.to_lowercase() == label_lower {
|
||||
logger::log_info(
|
||||
"NATIVE_RECORDER",
|
||||
&format!("Using selected native input device: {}", name),
|
||||
);
|
||||
return Ok(device);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(err) => {
|
||||
logger::log_error(
|
||||
"NATIVE_RECORDER",
|
||||
&format!("Failed to list native input devices: {}", err),
|
||||
);
|
||||
return Err(format!(
|
||||
"Не удалось найти выбранный микрофон для нативной записи: {}",
|
||||
err
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
return Err(format!(
|
||||
"Выбранный микрофон недоступен для нативной записи: {}",
|
||||
label
|
||||
));
|
||||
}
|
||||
|
||||
host.default_input_device()
|
||||
.ok_or_else(|| "Системный микрофон не найден.".to_string())
|
||||
}
|
||||
|
||||
fn append_samples(state: &Arc<Mutex<NativeRecorderState>>, channels: usize, samples: &[f32]) {
|
||||
if channels == 0 {
|
||||
return;
|
||||
}
|
||||
|
||||
let mut guard = match state.lock() {
|
||||
Ok(guard) => guard,
|
||||
Err(err) => err.into_inner(),
|
||||
};
|
||||
if guard.paused {
|
||||
return;
|
||||
}
|
||||
|
||||
guard.samples.reserve(samples.len() / channels);
|
||||
for frame in samples.chunks(channels) {
|
||||
let mut sum = 0.0;
|
||||
for sample in frame {
|
||||
sum += *sample;
|
||||
}
|
||||
guard.samples.push(sum / frame.len().max(1) as f32);
|
||||
}
|
||||
}
|
||||
|
||||
fn append_f32_samples(state: &Arc<Mutex<NativeRecorderState>>, channels: usize, data: &[f32]) {
|
||||
append_samples(state, channels, data);
|
||||
}
|
||||
|
||||
fn append_i16_samples(state: &Arc<Mutex<NativeRecorderState>>, channels: usize, data: &[i16]) {
|
||||
if channels == 0 {
|
||||
return;
|
||||
}
|
||||
|
||||
let mut guard = match state.lock() {
|
||||
Ok(guard) => guard,
|
||||
Err(err) => err.into_inner(),
|
||||
};
|
||||
if guard.paused {
|
||||
return;
|
||||
}
|
||||
|
||||
guard.samples.reserve(data.len() / channels);
|
||||
for frame in data.chunks(channels) {
|
||||
let mut sum = 0.0;
|
||||
for sample in frame {
|
||||
sum += *sample as f32 / i16::MAX as f32;
|
||||
}
|
||||
guard.samples.push(sum / frame.len().max(1) as f32);
|
||||
}
|
||||
}
|
||||
|
||||
fn append_u16_samples(state: &Arc<Mutex<NativeRecorderState>>, channels: usize, data: &[u16]) {
|
||||
if channels == 0 {
|
||||
return;
|
||||
}
|
||||
|
||||
let mut guard = match state.lock() {
|
||||
Ok(guard) => guard,
|
||||
Err(err) => err.into_inner(),
|
||||
};
|
||||
if guard.paused {
|
||||
return;
|
||||
}
|
||||
|
||||
guard.samples.reserve(data.len() / channels);
|
||||
for frame in data.chunks(channels) {
|
||||
let mut sum = 0.0;
|
||||
for sample in frame {
|
||||
sum += (*sample as f32 - 32_768.0) / 32_768.0;
|
||||
}
|
||||
guard.samples.push(sum / frame.len().max(1) as f32);
|
||||
}
|
||||
}
|
||||
|
||||
fn build_input_stream(
|
||||
device: &cpal::Device,
|
||||
config: &cpal::StreamConfig,
|
||||
sample_format: cpal::SampleFormat,
|
||||
state: Arc<Mutex<NativeRecorderState>>,
|
||||
) -> Result<cpal::Stream, String> {
|
||||
let channels = config.channels as usize;
|
||||
if channels == 0 {
|
||||
return Err("Микрофон вернул аудиоформат без каналов.".to_string());
|
||||
}
|
||||
|
||||
let err_fn = |err| {
|
||||
logger::log_error(
|
||||
"NATIVE_RECORDER",
|
||||
&format!("Native input stream error: {}", err),
|
||||
);
|
||||
};
|
||||
|
||||
match sample_format {
|
||||
cpal::SampleFormat::F32 => device
|
||||
.build_input_stream(
|
||||
config,
|
||||
move |data: &[f32], _| append_f32_samples(&state, channels, data),
|
||||
err_fn,
|
||||
None,
|
||||
)
|
||||
.map_err(|err| format!("Не удалось открыть микрофон: {}", err)),
|
||||
cpal::SampleFormat::I16 => device
|
||||
.build_input_stream(
|
||||
config,
|
||||
move |data: &[i16], _| append_i16_samples(&state, channels, data),
|
||||
err_fn,
|
||||
None,
|
||||
)
|
||||
.map_err(|err| format!("Не удалось открыть микрофон: {}", err)),
|
||||
cpal::SampleFormat::U16 => device
|
||||
.build_input_stream(
|
||||
config,
|
||||
move |data: &[u16], _| append_u16_samples(&state, channels, data),
|
||||
err_fn,
|
||||
None,
|
||||
)
|
||||
.map_err(|err| format!("Не удалось открыть микрофон: {}", err)),
|
||||
other => Err(format!(
|
||||
"Нативная запись пока не поддерживает формат микрофона {:?}.",
|
||||
other
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
fn resample_linear(input: &[f32], source_sample_rate: u32) -> Vec<f32> {
|
||||
if input.is_empty() || source_sample_rate == 0 {
|
||||
return Vec::new();
|
||||
}
|
||||
|
||||
if source_sample_rate == TARGET_SAMPLE_RATE {
|
||||
return input.to_vec();
|
||||
}
|
||||
|
||||
let ratio = source_sample_rate as f64 / TARGET_SAMPLE_RATE as f64;
|
||||
let output_len = ((input.len() as f64) / ratio).round().max(1.0) as usize;
|
||||
let mut output = Vec::with_capacity(output_len);
|
||||
|
||||
for output_index in 0..output_len {
|
||||
let source_pos = output_index as f64 * ratio;
|
||||
let index = source_pos.floor() as usize;
|
||||
let frac = (source_pos - index as f64) as f32;
|
||||
let current = input.get(index).copied().unwrap_or(0.0);
|
||||
let next = input.get(index + 1).copied().unwrap_or(current);
|
||||
output.push(current + (next - current) * frac);
|
||||
}
|
||||
|
||||
output
|
||||
}
|
||||
|
||||
fn normalize_to_i16(samples: &[f32]) -> (Vec<i16>, PcmStats) {
|
||||
if samples.is_empty() {
|
||||
return (
|
||||
Vec::new(),
|
||||
PcmStats {
|
||||
peak: 0.0,
|
||||
rms: 0.0,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
let mean = samples.iter().copied().sum::<f32>() / samples.len() as f32;
|
||||
let mut peak: f32 = 0.0;
|
||||
|
||||
for sample in samples {
|
||||
let centered = *sample - mean;
|
||||
peak = peak.max(centered.abs());
|
||||
}
|
||||
|
||||
let gain = if peak > PCM_MIN_SIGNAL_PEAK && peak < PCM_NORMALIZE_BELOW_PEAK {
|
||||
(PCM_TARGET_PEAK / peak).min(PCM_MAX_GAIN)
|
||||
} else {
|
||||
1.0
|
||||
};
|
||||
|
||||
let mut normalized_peak: f32 = 0.0;
|
||||
let mut normalized_sum_squares = 0.0_f64;
|
||||
let mut pcm = Vec::with_capacity(samples.len());
|
||||
for sample in samples {
|
||||
let normalized = ((*sample - mean) * gain).clamp(-1.0, 1.0);
|
||||
normalized_peak = normalized_peak.max(normalized.abs());
|
||||
normalized_sum_squares += (normalized as f64) * (normalized as f64);
|
||||
let value = if normalized < 0.0 {
|
||||
normalized * 32_768.0
|
||||
} else {
|
||||
normalized * 32_767.0
|
||||
};
|
||||
pcm.push(value.round() as i16);
|
||||
}
|
||||
|
||||
let rms = (normalized_sum_squares / samples.len() as f64).sqrt() as f32;
|
||||
(
|
||||
pcm,
|
||||
PcmStats {
|
||||
peak: normalized_peak,
|
||||
rms,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
fn write_wav_bytes(samples: &[i16]) -> Result<Vec<u8>, String> {
|
||||
let path = unique_temp_path("native-voice-recording", "wav");
|
||||
let spec = hound::WavSpec {
|
||||
channels: TARGET_CHANNELS,
|
||||
sample_rate: TARGET_SAMPLE_RATE,
|
||||
bits_per_sample: 16,
|
||||
sample_format: hound::SampleFormat::Int,
|
||||
};
|
||||
|
||||
let write_result = (|| -> Result<(), String> {
|
||||
let mut writer = hound::WavWriter::create(&path, spec)
|
||||
.map_err(|err| format!("Не удалось создать WAV запись: {}", err))?;
|
||||
for sample in samples {
|
||||
writer
|
||||
.write_sample(*sample)
|
||||
.map_err(|err| format!("Не удалось записать WAV sample: {}", err))?;
|
||||
}
|
||||
writer
|
||||
.finalize()
|
||||
.map_err(|err| format!("Не удалось завершить WAV запись: {}", err))
|
||||
})();
|
||||
|
||||
if let Err(err) = write_result {
|
||||
let _ = fs::remove_file(&path);
|
||||
return Err(err);
|
||||
}
|
||||
|
||||
let bytes = fs::read(&path).map_err(|err| {
|
||||
let _ = fs::remove_file(&path);
|
||||
format!("Не удалось прочитать WAV запись: {}", err)
|
||||
})?;
|
||||
let _ = fs::remove_file(&path);
|
||||
Ok(bytes)
|
||||
}
|
||||
|
||||
fn run_recorder_thread(
|
||||
req: StartNativeVoiceRecordingRequest,
|
||||
state: Arc<Mutex<NativeRecorderState>>,
|
||||
started_tx: mpsc::Sender<Result<NativeRecorderThreadInfo, String>>,
|
||||
stop_rx: mpsc::Receiver<()>,
|
||||
stopped_tx: mpsc::Sender<Result<(), String>>,
|
||||
) {
|
||||
let start_result = (|| -> Result<(cpal::Stream, NativeRecorderThreadInfo), String> {
|
||||
let host = cpal::default_host();
|
||||
let device = select_input_device(&host, req.device_label.as_deref())?;
|
||||
let device_name = device
|
||||
.name()
|
||||
.unwrap_or_else(|_| "default input".to_string());
|
||||
let supported_config = device
|
||||
.default_input_config()
|
||||
.map_err(|err| format!("Не удалось получить формат микрофона: {}", err))?;
|
||||
let sample_format = supported_config.sample_format();
|
||||
let config: cpal::StreamConfig = supported_config.into();
|
||||
let source_sample_rate = config.sample_rate.0;
|
||||
let source_channels = config.channels;
|
||||
let stream = build_input_stream(&device, &config, sample_format, Arc::clone(&state))?;
|
||||
|
||||
stream
|
||||
.play()
|
||||
.map_err(|err| format!("Не удалось запустить нативную запись: {}", err))?;
|
||||
|
||||
logger::log_info(
|
||||
"NATIVE_RECORDER",
|
||||
&format!(
|
||||
"Native voice recorder started: device={}, source_sample_rate={}, channels={}, sample_format={:?}",
|
||||
device_name, source_sample_rate, source_channels, sample_format
|
||||
),
|
||||
);
|
||||
|
||||
Ok((
|
||||
stream,
|
||||
NativeRecorderThreadInfo {
|
||||
source_sample_rate,
|
||||
source_channels,
|
||||
device_name,
|
||||
},
|
||||
))
|
||||
})();
|
||||
|
||||
let (stream, info) = match start_result {
|
||||
Ok(value) => value,
|
||||
Err(err) => {
|
||||
let _ = started_tx.send(Err(err));
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
if started_tx.send(Ok(info)).is_err() {
|
||||
drop(stream);
|
||||
return;
|
||||
}
|
||||
|
||||
let _ = stop_rx.recv();
|
||||
drop(stream);
|
||||
let _ = stopped_tx.send(Ok(()));
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
pub fn start_native_voice_recording(req: StartNativeVoiceRecordingRequest) -> Result<(), String> {
|
||||
let mut guard = recorder_slot()
|
||||
.lock()
|
||||
.map_err(|_| "Не удалось заблокировать нативную запись.".to_string())?;
|
||||
if guard.is_some() {
|
||||
return Err("Запись уже идёт.".to_string());
|
||||
}
|
||||
|
||||
let state = Arc::new(Mutex::new(NativeRecorderState::default()));
|
||||
let (started_tx, started_rx) = mpsc::channel();
|
||||
let (stop_tx, stop_rx) = mpsc::channel();
|
||||
let (stopped_tx, stopped_rx) = mpsc::channel();
|
||||
let thread_state = Arc::clone(&state);
|
||||
std::thread::Builder::new()
|
||||
.name("talkis-native-voice-recorder".to_string())
|
||||
.spawn(move || run_recorder_thread(req, thread_state, started_tx, stop_rx, stopped_tx))
|
||||
.map_err(|err| format!("Не удалось создать поток нативной записи: {}", err))?;
|
||||
|
||||
let info = started_rx
|
||||
.recv()
|
||||
.map_err(|_| "Нативная запись завершилась до запуска.".to_string())??;
|
||||
|
||||
*guard = Some(NativeVoiceRecorder {
|
||||
state,
|
||||
stop_tx,
|
||||
stopped_rx,
|
||||
source_sample_rate: info.source_sample_rate,
|
||||
source_channels: info.source_channels,
|
||||
device_name: info.device_name,
|
||||
});
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
pub fn pause_native_voice_recording() -> Result<(), String> {
|
||||
let guard = recorder_slot()
|
||||
.lock()
|
||||
.map_err(|_| "Не удалось заблокировать нативную запись.".to_string())?;
|
||||
let recorder = guard
|
||||
.as_ref()
|
||||
.ok_or_else(|| "Активная нативная запись не найдена.".to_string())?;
|
||||
let mut state = recorder
|
||||
.state
|
||||
.lock()
|
||||
.map_err(|_| "Не удалось поставить нативную запись на паузу.".to_string())?;
|
||||
state.paused = true;
|
||||
logger::log_info("NATIVE_RECORDER", "Native voice recorder paused");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
pub fn resume_native_voice_recording() -> Result<(), String> {
|
||||
let guard = recorder_slot()
|
||||
.lock()
|
||||
.map_err(|_| "Не удалось заблокировать нативную запись.".to_string())?;
|
||||
let recorder = guard
|
||||
.as_ref()
|
||||
.ok_or_else(|| "Активная нативная запись не найдена.".to_string())?;
|
||||
let mut state = recorder
|
||||
.state
|
||||
.lock()
|
||||
.map_err(|_| "Не удалось продолжить нативную запись.".to_string())?;
|
||||
state.paused = false;
|
||||
logger::log_info("NATIVE_RECORDER", "Native voice recorder resumed");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
pub fn stop_native_voice_recording() -> Result<NativeVoiceRecordingResult, String> {
|
||||
let recorder = recorder_slot()
|
||||
.lock()
|
||||
.map_err(|_| "Не удалось заблокировать нативную запись.".to_string())?
|
||||
.take()
|
||||
.ok_or_else(|| "Активная нативная запись не найдена.".to_string())?;
|
||||
|
||||
let source_sample_rate = recorder.source_sample_rate;
|
||||
let source_channels = recorder.source_channels;
|
||||
let device_name = recorder.device_name.clone();
|
||||
let state = Arc::clone(&recorder.state);
|
||||
recorder
|
||||
.stop_tx
|
||||
.send(())
|
||||
.map_err(|_| "Не удалось остановить поток нативной записи.".to_string())?;
|
||||
match recorder.stopped_rx.recv_timeout(Duration::from_secs(3)) {
|
||||
Ok(Ok(())) => {}
|
||||
Ok(Err(err)) => return Err(err),
|
||||
Err(mpsc::RecvTimeoutError::Timeout) => {
|
||||
return Err("Нативная запись не остановилась вовремя.".to_string());
|
||||
}
|
||||
Err(mpsc::RecvTimeoutError::Disconnected) => {
|
||||
logger::log_error(
|
||||
"NATIVE_RECORDER",
|
||||
"Native recorder thread ended before stop acknowledgement",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
let source_samples = {
|
||||
let mut guard = state
|
||||
.lock()
|
||||
.map_err(|_| "Не удалось прочитать нативную запись.".to_string())?;
|
||||
std::mem::take(&mut guard.samples)
|
||||
};
|
||||
let resampled = resample_linear(&source_samples, source_sample_rate);
|
||||
let (pcm_samples, stats) = normalize_to_i16(&resampled);
|
||||
let wav_bytes = write_wav_bytes(&pcm_samples)?;
|
||||
let duration_ms =
|
||||
((pcm_samples.len() as f64 / TARGET_SAMPLE_RATE as f64) * 1000.0).round() as u64;
|
||||
|
||||
logger::log_info(
|
||||
"NATIVE_RECORDER",
|
||||
&format!(
|
||||
"Native voice recorder stopped: device={}, source_sample_rate={}, source_channels={}, source_samples={}, duration_ms={}, sample_rate={}, channels={}, peak={:.4}, rms={:.4}",
|
||||
device_name,
|
||||
source_sample_rate,
|
||||
source_channels,
|
||||
source_samples.len(),
|
||||
duration_ms,
|
||||
TARGET_SAMPLE_RATE,
|
||||
TARGET_CHANNELS,
|
||||
stats.peak,
|
||||
stats.rms
|
||||
),
|
||||
);
|
||||
|
||||
Ok(NativeVoiceRecordingResult {
|
||||
audio_base64: base64::engine::general_purpose::STANDARD.encode(wav_bytes),
|
||||
mime_type: "audio/wav".to_string(),
|
||||
file_name: "recording.wav".to_string(),
|
||||
duration_ms,
|
||||
sample_rate: TARGET_SAMPLE_RATE,
|
||||
channels: TARGET_CHANNELS,
|
||||
peak: stats.peak,
|
||||
rms: stats.rms,
|
||||
})
|
||||
}
|
||||
|
|
@ -76,8 +76,8 @@ fn get_linux_active_window() -> Result<Option<LinuxPasteTarget>, String> {
|
|||
use x11rb::connection::Connection;
|
||||
use x11rb::protocol::xproto::{AtomEnum, ConnectionExt};
|
||||
|
||||
let (conn, screen_num) = x11rb::connect(None)
|
||||
.map_err(|err| format!("Failed to connect to X11: {}", err))?;
|
||||
let (conn, screen_num) =
|
||||
x11rb::connect(None).map_err(|err| format!("Failed to connect to X11: {}", err))?;
|
||||
let screen = &conn.setup().roots[screen_num];
|
||||
let active_window_atom = conn
|
||||
.intern_atom(false, b"_NET_ACTIVE_WINDOW")
|
||||
|
|
@ -178,7 +178,10 @@ pub fn remember_linux_paste_target_window() {
|
|||
logger::log_info("PASTE", "No active Linux paste target window to remember");
|
||||
}
|
||||
Err(err) => {
|
||||
logger::log_error("PASTE", &format!("Failed to remember paste target: {}", err));
|
||||
logger::log_error(
|
||||
"PASTE",
|
||||
&format!("Failed to remember paste target: {}", err),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -224,7 +227,10 @@ fn write_linux_xclip_text(text: &str) {
|
|||
match write_linux_xclip_selection(selection, text) {
|
||||
Ok(()) => logger::log_info(
|
||||
"PASTE",
|
||||
&format!("Wrote recognized text to X11 {} selection via xclip", selection),
|
||||
&format!(
|
||||
"Wrote recognized text to X11 {} selection via xclip",
|
||||
selection
|
||||
),
|
||||
),
|
||||
Err(err) => logger::log_error("PASTE", &err),
|
||||
}
|
||||
|
|
@ -249,8 +255,8 @@ fn focus_remembered_linux_paste_target() {
|
|||
let window = target.window;
|
||||
|
||||
let result = (|| -> Result<(), String> {
|
||||
let (conn, screen_num) = x11rb::connect(None)
|
||||
.map_err(|err| format!("Failed to connect to X11: {}", err))?;
|
||||
let (conn, screen_num) =
|
||||
x11rb::connect(None).map_err(|err| format!("Failed to connect to X11: {}", err))?;
|
||||
let screen = &conn.setup().roots[screen_num];
|
||||
let active_window_atom = conn
|
||||
.intern_atom(false, b"_NET_ACTIVE_WINDOW")
|
||||
|
|
@ -288,7 +294,10 @@ fn focus_remembered_linux_paste_target() {
|
|||
|
||||
match result {
|
||||
Ok(()) => {
|
||||
logger::log_info("PASTE", &format!("Focused remembered Linux paste target: {}", window));
|
||||
logger::log_info(
|
||||
"PASTE",
|
||||
&format!("Focused remembered Linux paste target: {}", window),
|
||||
);
|
||||
std::thread::sleep(Duration::from_millis(180));
|
||||
}
|
||||
Err(err) => logger::log_error("PASTE", &err),
|
||||
|
|
@ -393,8 +402,8 @@ fn simulate_cmd_v() -> Result<(), String> {
|
|||
let warp_paste = should_use_warp_paste_shortcut();
|
||||
let terminal_paste = should_use_terminal_paste_shortcut();
|
||||
|
||||
let (conn, screen_num) = x11rb::connect(None)
|
||||
.map_err(|err| format!("Failed to connect to X11: {}", err))?;
|
||||
let (conn, screen_num) =
|
||||
x11rb::connect(None).map_err(|err| format!("Failed to connect to X11: {}", err))?;
|
||||
let root = conn.setup().roots[screen_num].root;
|
||||
|
||||
let events: &[(u8, u8)] = if warp_paste {
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
"$schema": "https://schema.tauri.app/config/2",
|
||||
"productName": "Talkis",
|
||||
"mainBinaryName": "Talkis",
|
||||
"version": "0.1.21",
|
||||
"version": "0.1.22",
|
||||
"identifier": "com.trixter.talkis",
|
||||
"build": {
|
||||
"beforeDevCommand": "bun run prepare:sidecars && bun run dev",
|
||||
|
|
|
|||
|
|
@ -57,6 +57,21 @@ function getAudioConstraints(micId: string): MediaTrackConstraints | true {
|
|||
return constraints;
|
||||
}
|
||||
|
||||
async function resolveSelectedMicLabel(micId: string): Promise<string | null> {
|
||||
if (!micId || !navigator.mediaDevices?.enumerateDevices) {
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
const devices = await navigator.mediaDevices.enumerateDevices();
|
||||
const selected = devices.find((device) => device.kind === "audioinput" && device.deviceId === micId);
|
||||
return selected?.label?.trim() || null;
|
||||
} catch (error) {
|
||||
logError("RECORDING", `Failed to resolve selected mic label: ${formatErrorMessage(error)}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function waitForTrackReady(stream: MediaStream, timeoutMs: number): Promise<void> {
|
||||
const [track] = stream.getAudioTracks();
|
||||
if (!track || (!track.muted && track.readyState === "live")) {
|
||||
|
|
@ -234,6 +249,30 @@ export function useWidgetRecording({
|
|||
void resizeWidget(CALL_STACK_WIDGET_WIDTH, CALL_STACK_WIDGET_HEIGHT);
|
||||
|
||||
recordingSettingsRef.current = activeSettings;
|
||||
const nativeMicLabel = await resolveSelectedMicLabel(activeSettings.micId);
|
||||
if (activeSettings.micId && nativeMicLabel) {
|
||||
logInfo("RECORDING", `Using preferred native mic label: ${nativeMicLabel}`);
|
||||
} else if (activeSettings.micId) {
|
||||
logInfo("RECORDING", "Selected mic label is unavailable for native recorder; using WebView recorder to preserve selected mic");
|
||||
}
|
||||
|
||||
if (!activeSettings.micId || nativeMicLabel) {
|
||||
try {
|
||||
const codec = await runtimeRef.current.startNative({ deviceLabel: nativeMicLabel });
|
||||
logInfo("RECORDING", codec === "native-wav" ? "Using native wav recorder" : "Using native recorder");
|
||||
setStream(null);
|
||||
logInfo("RECORDING", "Recording started successfully");
|
||||
dispatch({ type: "RECORDING_STARTED", timestamp: Date.now() });
|
||||
return;
|
||||
} catch (nativeError) {
|
||||
runtimeRef.current.reset();
|
||||
logError(
|
||||
"RECORDING",
|
||||
`Native recorder start failed, falling back to WebView recorder: ${formatErrorMessage(nativeError)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const audioConstraints = getAudioConstraints(activeSettings.micId);
|
||||
if (activeSettings.micId) {
|
||||
logInfo("RECORDING", `Using preferred mic: ${activeSettings.micId}`);
|
||||
|
|
@ -326,9 +365,9 @@ export function useWidgetRecording({
|
|||
void resizeWidget(CALL_STACK_WIDGET_WIDTH, CALL_STACK_WIDGET_HEIGHT);
|
||||
await waitForWidgetPaint();
|
||||
|
||||
await runtimeRef.current.stop();
|
||||
|
||||
try {
|
||||
await runtimeRef.current.stop();
|
||||
|
||||
if (!runtimeRef.current.hasAudioChunks()) {
|
||||
logError("RECORDING", "No audio chunks recorded");
|
||||
throw new Error("Аудио не записано. Попробуйте еще раз.");
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
import { logInfo } from "../../../lib/logger";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
|
||||
type RecorderCodec = "webm" | "default" | "wav";
|
||||
import { logError, logInfo } from "../../../lib/logger";
|
||||
|
||||
type RecorderCodec = "native-wav" | "webm" | "default" | "wav";
|
||||
|
||||
const PCM_TARGET_PEAK = 0.82;
|
||||
const PCM_NORMALIZE_BELOW_PEAK = 0.35;
|
||||
|
|
@ -19,6 +21,8 @@ interface PcmRecorderState {
|
|||
|
||||
interface RecordingRuntimeState {
|
||||
active: boolean;
|
||||
nativeActive: boolean;
|
||||
nativeResult: NativeVoiceRecordingResult | null;
|
||||
recorder: MediaRecorder | null;
|
||||
chunks: Blob[];
|
||||
stream: MediaStream | null;
|
||||
|
|
@ -41,7 +45,23 @@ interface EncodedWavResult {
|
|||
stats: PcmAudioStats;
|
||||
}
|
||||
|
||||
interface NativeRecordingOptions {
|
||||
deviceLabel?: string | null;
|
||||
}
|
||||
|
||||
interface NativeVoiceRecordingResult {
|
||||
audioBase64: string;
|
||||
mimeType: string;
|
||||
fileName: string;
|
||||
durationMs: number;
|
||||
sampleRate: number;
|
||||
channels: number;
|
||||
peak: number;
|
||||
rms: number;
|
||||
}
|
||||
|
||||
export interface RecordingRuntimeController {
|
||||
startNative(options?: NativeRecordingOptions): Promise<RecorderCodec>;
|
||||
start(stream: MediaStream): RecorderCodec;
|
||||
pause(): boolean;
|
||||
resume(): boolean;
|
||||
|
|
@ -81,6 +101,15 @@ function stopTracks(stream: MediaStream | null): void {
|
|||
stream?.getTracks().forEach((track) => track.stop());
|
||||
}
|
||||
|
||||
function base64ToBlob(base64: string, mimeType: string): Blob {
|
||||
const binary = atob(base64);
|
||||
const bytes = new Uint8Array(binary.length);
|
||||
for (let i = 0; i < binary.length; i += 1) {
|
||||
bytes[i] = binary.charCodeAt(i);
|
||||
}
|
||||
return new Blob([bytes], { type: mimeType });
|
||||
}
|
||||
|
||||
function shouldUsePcmOnlyRecorder(): boolean {
|
||||
if (typeof navigator === "undefined") {
|
||||
return false;
|
||||
|
|
@ -352,6 +381,8 @@ function encodeWavInWorker(chunks: Float32Array[], sampleRate: number): Promise<
|
|||
export function createRecordingRuntimeController(): RecordingRuntimeController {
|
||||
const state: RecordingRuntimeState = {
|
||||
active: false,
|
||||
nativeActive: false,
|
||||
nativeResult: null,
|
||||
recorder: null,
|
||||
chunks: [],
|
||||
stream: null,
|
||||
|
|
@ -360,9 +391,36 @@ export function createRecordingRuntimeController(): RecordingRuntimeController {
|
|||
};
|
||||
|
||||
return {
|
||||
async startNative(options) {
|
||||
state.chunks = [];
|
||||
state.nativeResult = null;
|
||||
state.stream = null;
|
||||
state.recorder = null;
|
||||
stopPcmRecorder(state.pcm);
|
||||
state.pcm = createEmptyPcmState();
|
||||
state.pcmOnly = false;
|
||||
|
||||
try {
|
||||
await invoke("start_native_voice_recording", {
|
||||
req: {
|
||||
deviceLabel: options?.deviceLabel || null,
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
state.active = false;
|
||||
state.nativeActive = false;
|
||||
throw error;
|
||||
}
|
||||
|
||||
state.active = true;
|
||||
state.nativeActive = true;
|
||||
return "native-wav";
|
||||
},
|
||||
start(stream) {
|
||||
state.stream = stream;
|
||||
state.chunks = [];
|
||||
state.nativeActive = false;
|
||||
state.nativeResult = null;
|
||||
state.pcm = startPcmRecorder(stream);
|
||||
state.active = true;
|
||||
state.pcmOnly = shouldUsePcmOnlyRecorder();
|
||||
|
|
@ -396,6 +454,13 @@ export function createRecordingRuntimeController(): RecordingRuntimeController {
|
|||
return codec;
|
||||
},
|
||||
pause() {
|
||||
if (state.nativeActive && state.active) {
|
||||
void invoke("pause_native_voice_recording").catch((error) => {
|
||||
logError("RECORDING", `Native recorder pause failed: ${error instanceof Error ? error.message : String(error)}`);
|
||||
});
|
||||
return true;
|
||||
}
|
||||
|
||||
if (state.pcmOnly && state.active) {
|
||||
state.pcm.paused = true;
|
||||
return true;
|
||||
|
|
@ -415,6 +480,13 @@ export function createRecordingRuntimeController(): RecordingRuntimeController {
|
|||
}
|
||||
},
|
||||
resume() {
|
||||
if (state.nativeActive && state.active) {
|
||||
void invoke("resume_native_voice_recording").catch((error) => {
|
||||
logError("RECORDING", `Native recorder resume failed: ${error instanceof Error ? error.message : String(error)}`);
|
||||
});
|
||||
return true;
|
||||
}
|
||||
|
||||
if (state.pcmOnly && state.active) {
|
||||
state.pcm.paused = false;
|
||||
return true;
|
||||
|
|
@ -437,6 +509,13 @@ export function createRecordingRuntimeController(): RecordingRuntimeController {
|
|||
return;
|
||||
}
|
||||
|
||||
if (state.nativeActive) {
|
||||
state.nativeResult = await invoke<NativeVoiceRecordingResult>("stop_native_voice_recording");
|
||||
state.active = false;
|
||||
state.nativeActive = false;
|
||||
return;
|
||||
}
|
||||
|
||||
if (state.pcmOnly || !state.recorder) {
|
||||
stopTracks(state.stream);
|
||||
state.stream = null;
|
||||
|
|
@ -460,9 +539,22 @@ export function createRecordingRuntimeController(): RecordingRuntimeController {
|
|||
return state.active;
|
||||
},
|
||||
hasAudioChunks() {
|
||||
if (state.nativeResult) {
|
||||
return state.nativeResult.audioBase64.length > 0;
|
||||
}
|
||||
|
||||
return state.chunks.length > 0 || pcmSampleCount(state.pcm.chunks) > 0;
|
||||
},
|
||||
async getAudioBlob() {
|
||||
if (state.nativeResult) {
|
||||
const result = state.nativeResult;
|
||||
logInfo(
|
||||
"RECORDING",
|
||||
`Native audio stats: duration_ms=${result.durationMs}, sample_rate=${result.sampleRate}, channels=${result.channels}, peak=${result.peak.toFixed(4)}, rms=${result.rms.toFixed(4)}`,
|
||||
);
|
||||
return base64ToBlob(result.audioBase64, result.mimeType || "audio/wav");
|
||||
}
|
||||
|
||||
if (state.chunks.length > 0) {
|
||||
const type = state.chunks[0]?.type || "audio/webm";
|
||||
return new Blob(state.chunks, { type });
|
||||
|
|
@ -477,16 +569,23 @@ export function createRecordingRuntimeController(): RecordingRuntimeController {
|
|||
},
|
||||
reset() {
|
||||
state.active = false;
|
||||
state.nativeActive = false;
|
||||
state.nativeResult = null;
|
||||
state.recorder = null;
|
||||
state.chunks = [];
|
||||
state.pcm = createEmptyPcmState();
|
||||
state.pcmOnly = false;
|
||||
},
|
||||
dispose() {
|
||||
if (state.nativeActive) {
|
||||
void invoke("stop_native_voice_recording").catch(() => null);
|
||||
}
|
||||
stopPcmRecorder(state.pcm);
|
||||
stopTracks(state.stream);
|
||||
state.stream = null;
|
||||
state.active = false;
|
||||
state.nativeActive = false;
|
||||
state.nativeResult = null;
|
||||
state.recorder = null;
|
||||
state.chunks = [];
|
||||
state.pcm = createEmptyPcmState();
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue