mirror of
https://github.com/cogwheel0/conduit.git
synced 2026-10-14 03:33:01 +00:00
9ee7b114ee
* Keep what a voice call said on its messages, and read Open WebUI's realtime call settings Open WebUI 0.12 stores a realtime call's spoken transcripts in each message's meta.voice and replays them to the chat model. A message Conduit wrote again from its own state carried no meta, and the server replaces each message whole, so the next push would have erased them. Rows rewritten locally now keep the meta stored on them, the voice record is read into the message's metadata, and the echo writes it back where Open WebUI reads it. Another device's chat:message:voice refreshes the chat. Direct chats replay what a call spoke the way Open WebUI does: a reply the voice gave by itself becomes its transcript, labeled as historical voice speech, and what was spoken over a chat answer follows that answer. The config's audio.realtime block and the chat.call permission are read, ready for gating realtime calls; unreadable permissions count as unknown. * Speak and listen through each chat's own server, including Hermes and Direct The Server engines for speech to text and text to speech only worked with an Open WebUI server, and in a Hermes or Direct chat they quietly used whichever Open WebUI server was signed in. Server speech now follows the selected model's backend: an Open WebUI model uses its server, a Hermes chat on a Desktop Gateway uses the gateway's audio relay, and a Direct or Apple chat uses a new Voice provider. A chat whose backend has none uses the device instead; audio never goes to a server the chat does not belong to. Hermes speech goes through /api/audio/transcribe and /api/audio/speak for the connection's profile, with the providers and keys configured on the gateway; the voice-config route, which hands out provider keys, is never called. The Voice provider is one Direct connection that speaks the OpenAI API, chosen in Audio settings with its transcription model, speech model and voice. Requests go to its audio/transcriptions and audio/speech with the connection's own key, headers, api-version and certificates. Picking OpenAI fills in the models Open WebUI defaults to. The page is also in the native Settings sheet on iOS. The server voice picker shows only for Open WebUI, which lists voices; Hermes and the Voice provider speak with their own. * Talk to a realtime voice in calls, ready for Open WebUI and Direct Open WebUI 0.12 can run calls through an OpenAI realtime voice that answers small talk itself and hands real requests to the chat's model. This adds the call side of that, shared by Open WebUI and Direct chats; the audio engine it needs comes next, so calls stay as they are until then. The engine follows Open WebUI's web client: one request to the voice at a time, made only while nobody is speaking; the chat's latest state sent before each; speaking over the voice cancels its reply and tells it how much was heard; a request the voice delegates runs as an ordinary chat turn, and its outcome goes back to be spoken. Open WebUI's server bridge carries a call for its chats. A Direct or Apple chat talks to the Voice provider's Realtime API instead, with the bridge's rules checked on the device, so one engine runs both; the device's bridge is tested against the server's own rules, and the engine's tests hold every command to them. A delegated request is sent like a typed message, with the user's words marked as spoken. A reply the voice gives itself is stored through the same send as a finished answer under the voice model, and no model runs; Open WebUI replays it to the chat model as past speech. What the voice says about a delegated answer is merged into that answer's meta.voice. Words nothing answered stay in the call's captions. The call controller runs either engine: a call with a realtime voice keeps CallKit, the audio session, background work, mute, speaker and the call panel as before. Whether a call has one is decided when it starts, and it falls back to a Standard call whenever anything is unknown: Standard chosen in settings, a temporary chat, a server without realtime calls, an account that may not call or whose permissions could not be read, a sign-in with an API key, or a Voice provider without a realtime model, voice and transcription model. Direct connections now share their key, header and certificate setup between HTTP and WebSocket, and Open WebUI's socket headers are built in one place, as is the WebSocket that never follows redirects. * Translate the Voice provider settings into Polish Polish carries every English key, so the Voice provider's strings need their Polish copies; without them polish_localization_test fails. * Talk with a realtime voice in calls on iOS and Android Calls in Direct and Apple chats with a Voice provider, and in Open WebUI chats on a server with realtime calls, now talk with a realtime voice: both sides can speak at once, and speaking over the voice stops it. The voice needs the microphone and the speaker on one echo-cancelled stream, so it does not hear itself. On iOS one audio engine runs with voice processing on: the microphone is converted to 24 kHz frames, and the voice plays from a queue that works like Open WebUI's web worklet, starting once 80 ms are buffered and reporting how much of each part was played when the user cuts in. It restarts after a route change. On Android the microphone records as voice communication with the platform's echo canceller, noise suppressor and gain control, and the voice plays on the same audio session; what was heard comes from the track's play position. A microphone that only offers 48 kHz is brought down to 24 kHz. If the audio stops mid-call, the call ends and says so. Audio settings gain Calls: Realtime, the default, or Standard, which keeps every call turn by turn. The Voice provider takes a realtime model, voice and optional instructions, prefilled for OpenAI. Both are in the native Settings sheet on iOS too. * Tell the user what a realtime call could not do, and mark what its voice said A realtime call's problems that do not end it now show as messages in the user's language: words that could not be made out, a reply cut short, and a part of the call that could not be saved to the chat. A reply the voice gave by itself shows its voice model with a waveform and "spoken", even when the answer above came from the same model, and offers no Regenerate: there is no chat model answer to run again. Opening another chat, or none, ends a realtime call: its voice talks about the chat it started in, and a request it handed over is answered there. A call started without a chat follows the chat its first turn creates, and that chat's server id. * Talk with Hermes's GPT-Live voice in calls on a Desktop Gateway A Hermes profile that runs voice calls through GPT-Live, and has an OpenAI key for it, now gets realtime calls in Conduit too. The gateway keeps the key: Conduit makes a WebRTC offer, Hermes opens the session with OpenAI and returns the answer, and the voice's audio and events then flow between the phone and OpenAI. Profiles in the chained voice mode, a gateway that cannot say, and Responses API connections keep Standard calls. The call follows Hermes's own desktop client. GPT-Live takes its own turns; a request it delegates carries no text, so Conduit sends the user's latest words as an ordinary Hermes turn, marked as coming from voice-live, with the spoken conversation of the last five minutes as context only the model reads. While Hermes works the voice is told so quietly; an approval waiting in the chat is mentioned once; the answer is handed back in sentence-sized pieces for the voice to say. A newer request stops the one still running, mute reaches the session, and ending a call closes the session before the connection. The WebRTC call comes from flutter_webrtc, behind the same port as the other call audio, so the call controller, CallKit, background work and the call panel are shared with every other call. * Translate realtime calls and the Voice provider into every language The Calls setting, the Voice provider page, the call's notices and the "spoken" label were in English and Polish only, and the Localization check requires every key in every language. They are now in all of them. Every language also still said Server speech always goes to Open WebUI. It now follows the chat's own server, so the descriptions name Open WebUI, Hermes, or the Voice provider for Direct chats, as the English does. * Keep the Voice provider page and the iOS call audio within the app's rules The Voice provider page imported the Material and Cupertino libraries directly, which the platform UI check allows only behind the shared facade; it now takes its icons from the shared vocabulary. The package boundary check also lists the new iOS call audio bridge, which reaches its host only through ConduitBridgeHost. * Fix what review found in realtime calls - A call writes only to its own chat. What it said could land in the chat the user opened as the call ended, and a call started without a chat adopted any chat opened next. It now follows only the chat its own send creates and that chat's server id, writes nothing elsewhere, and merges a delegated answer's transcript into the call's chat even after a switch. - A call that starts a new chat no longer seeds GPT-Live with the previous chat's messages, or names that chat to Open WebUI. - Hanging up while a call is starting leaves no microphone, WebRTC peer or socket open. A WebSocket handshake that times out is stopped, and a socket that opens after it is closed. - GPT-Live's microphone goes quiet before the session is asked to close, and an answer handed over as the call ends goes on in the chat unfollowed. - Muting while speaking no longer holds a finished answer back until the user speaks again. - Small talk held behind a delegated answer is saved after that answer when the call ends, so the answer stays last while it runs and can still be stopped. If it is still running a few seconds later, the small talk stays in the captions. - A delegated answer whose send failed after it was placed ends as failed instead of working forever. Hermes approvals and questions now count as waiting for the user, so the voice says to answer them in the chat. - GPT-Live's request is the user's latest words from the last five minutes, as in Hermes's own client; words from long before no longer join it. - Read-aloud and dictation keep the server they started with, so text and recordings never go to another chat's server when the model changes. - A voice reply's "spoken" label appears when its transcript arrives after the message, the native Voice provider choice is kept while connections reload, the call mode is saved on its own key, a Direct voice turn releases its database lease if saving it fails, and ending the call controller ends its realtime voice. * Fix what review found in the realtime call audio on iOS and Android - Android: a piece of the voice's speech taken just before the user cut in could still play, and count as heard. It is now taken with its clear generation, so a clear right after drops it. A stopped call's capture and playback threads stop with it, and an old thread sends nothing to the next call; the recorder is stopped before its thread is awaited, so a blocked read returns first. - iOS: after a Bluetooth or headset route change, the microphone tap and converter are rebuilt for the new route's format. The playback queue no longer builds a string on the audio thread for every render. * Fix what the second review found in realtime calls - The first request in a new chat no longer ends the call: the call's own send counts as creating its chat until it is done, as the chat can open only after the answer is placed. - A chat opened while the call connects ends it, and the voice never reads a chat other than the call's own. - A reply cut off by hanging up waits behind a delegated answer still running, like the small talk held before it. - A GPT-Live request replaced while the one before it was stopping is dropped instead of being sent to the chat. - Resetting settings clears the Voice provider for good; it came back on the next launch. - An earlier version of a voice reply is not labeled as spoken: versions keep no transcript to tell. - The call's chat host closes even when the voice's ending fails, and chunkForCommentary takes only a positive limit. * Follow only the chat a realtime call's own send created A call started without a chat followed any chat that opened while its first send was running, so a chat the user opened during that wait became the call's: the voice read it, and later requests went there. The call now remembers the messages its sends placed and follows a new chat only if it holds them, which every send path's new chat does when it opens. * Hand GPT-Live's newest request to the chat only once older answers stop A request replaced while the one before it was still stopping skipped the wait: the newest went to the chat while the first answer still ran, and the chat put it off as busy. Stopping replaced requests now runs as one chain, and a new request waits for all of it, including stops added while it waited. * Leave answers handed over as a realtime call ends running in the chat - A request handed to the chat as the call ended was stopped by the bridge engine, though the chat should go on answering it. Both engines now stop only following such an answer, through a new DelegatedTurn.close that leaves the answer running, and its listener no longer outlives the call. - A GPT-Live request still being handed to the chat when a newer one came is now stopped once the chat has it, before the newer request goes, so the chat never takes two at once. * Name Apple chats where Server speech says it uses the Voice provider The Voice provider page says Direct and Apple chats use it, but the Server speech descriptions named only Direct chats. Both now say Direct and Apple chats, in every language, worded as each language's Voice provider description already words it. * Keep realtime calls working alongside enforced temporary chats Open WebUI 0.12 support (#843) made a new chat start temporary when the server enforces it, and reading that now reaches the signed-in account and its permissions. Three things follow for realtime calls: - A call that starts a new chat stays Standard when the server will keep that chat temporary, as it already did for a temporary chat that is open. - A realtime route that cannot be read keeps the call Standard instead of failing it, like anything else the route cannot tell. - The Hermes route tests say the server enforces nothing, as they never sign in to Open WebUI. * End a realtime call at once when it is hung up while connecting - Hanging up while a realtime voice was still connecting waited for the start queued ahead of the stop, so the microphone and connection could stay up until the connection was made or timed out. The stop now ends a connecting voice right away, and the start it cuts short counts as cancelled, not failed. - A voice reply's "spoken" label kept its old language after the app's language changed: the cached header now keys on the label as shown. * Let every end of a realtime call wait for its one ending A hang-up while the voice connects ends it at once, and the teardown that follows ends it again. The second end returned as soon as the voice was marked ended, so teardown could turn off the audio session before the voice had stopped, and an error stopping it went unreported. Both engines now keep their ending: every end waits for the same one, and teardown reports what it throws. * End a connecting realtime call at once when the system call UI hangs up Hanging up from the CallKit screen queued the stop without the immediate end a hang-up in the app gets, so a realtime voice still connecting kept its microphone and connection until it connected or timed out, and could connect after the call had ended. Both hang-ups now share that immediate end, and the system's own end of the call is still not repeated. * Fix what Greptile's full review found in realtime calls - A chat opened while a call was still setting up kept the call going with the voice made for the chat the user left. The call now notes its chat when it starts, and a different chat open by the time it connects cancels it. - Stopping a request the call replaces now waits for the chat's stop to settle, so the stop's late effects cannot reach the new request. - An answer whose send failed after it was placed is marked failed in the chat, the way a typed send's is, instead of staying shown as running and keeping the chat busy. - Small talk waits while the chat is answering, whoever asked; it used to wait only for answers the call had delegated, so a typed answer could stop being the chat's last message and could no longer be stopped. - Pause holds the voice's answers until the call resumes, instead of only muting the microphone while finished answers went on being spoken. - Speech to text set to Server uses the device in a chat whose backend cannot transcribe, as the Voice provider promised, instead of hiding voice input. - Choosing the native Voice provider's connection, or editing its fields, now redraws that page, which iOS redraws only when it is the page the update is for. * Hold a realtime call to the chat it was asked for, and unpause on unmute - The chat a call is for is now noted when the call is asked for, before the setup that waits for the microphone and the route. A chat opened during that wait cancels the call instead of becoming its chat. - Unmuting a paused realtime call, from the app or CallKit, now resumes it too; the voice stayed paused and kept its answers back while the call showed as live. * Keep held small talk from silencing requests, and pause notices honest - Small talk said while a typed answer runs waited on the call's own writes, which a request then waited behind for minutes, so the caller heard nothing instead of being told the chat was busy. It is now held apart: a request meanwhile hears at once that the chat is busy, and one made after the answer comes after the held words. - A GPT-Live request that waited for approval while the call was paused, then finished, no longer has the voice say on resume that it still waits; its answer is still said. - The native Voice provider page is patched alone and at once when it opens, rather than after the speech voices it does not show have loaded, so the patch cannot land over text the user is typing. * Leave GPT-Live answers alone as a call closes, and drop replaced notices - A request the voice handed over while the call was closing replaced the answer still running, which stopped it although an ended call leaves its answer to go on in the chat. Such a request now replaces nothing. - A request replaced while the call was paused had its held approval notice said on resume, as if it still waited. Replacing a request now drops what was held about it. |
||
|---|---|---|
| .. | ||
| conduit_core | ||
| conduit_ddgs | ||
| conduit_markdown | ||
| conduit_theme | ||