Voice in the widget: microphone, speech and browsers

For developers, in code: what the browser needs, conversation mode versus dictation, the spoken-language chip, barge-in on desktop and half-duplex on a phone, iOS autoplay, and what a page can configure.

The widget has one voice tier, and both halves of it run on the server: a spoken answer is synthesized by POST /tts one sentence at a time and played from one shared <audio> element; recognition is the widget’s own microphone capture streamed to WS /ws/stt. There is no browser speech API involved and no fallback to one — voiceProvider is not a configuration key.

In the panel: Widget → Voice — Voice conversation, Voice input (dictation), Speech speed and a voice per language (Widget settings).

What the browser needs

For	Needs	Without it
spoken answers	`fetch` and `Audio`	text answers only
the microphone	`getUserMedia`, `AudioContext`, `WebSocket`, a secure origin	no microphone at all; text chat unaffected

Nothing plays sound or opens a microphone until the visitor presses something: the microphone press is the browser gesture that asks for the device and unlocks audio.

Two independent switches

Switch	What it gives the visitor
`conversationModeEnabled`	the character’s microphone in the dock: one press, then they talk and the assistant answers out loud and keeps listening — a voice conversation
`voiceInputEnabled`	the composer’s microphone: dictate a question into the text field; the answer comes as text

Either, both, or neither. A spoken answer exists only inside a voice conversation; a question typed during one is answered in text, and the conversation stays live. Features: voice says how it feels; this page says what it is.

How a conversation runs

Listening streams audio to /ws/stt; a final transcript becomes the message, sent automatically. Fillers («hmm», «um») are dropped client-side; a final whose word confidence is under the server’s floor is blanked, so noise never becomes a question.
Speaking queues the whole answer at done, split into sentences, and prefetches one sentence ahead so there is no gap; a sentence may start playing earlier when the server marks the stream speakable.
On a pointer device the visitor can interrupt by talking: echo cancellation keeps the microphone live during the answer, the first words of a new utterance cut the readout, and the assistant’s own voice is filtered out as echo.
On a touch device the conversation is half-duplex: the loudspeaker would trip the microphone, so the microphone is gated while a clip sounds. What interrupts is the ■ stop control (drawn while an answer sounds), a tap on the launcher, typing, or the mic press (which also pauses).
Pause and resume go through the mic press; focusing any text field auto-mutes. After 30 s of silence the conversation ends.
iOS grants audio per document and only after a tap: a remembered conversation comes back at rest, and the first tap on the launcher resumes it live. Expected, not a bug.

The spoken-language chip

A visitor does not pick a language; they talk. The widget guesses from the browser’s languages and the timezone’s region, streams that language, and after two consistent finals pins it. The chip beside the microphone shows which language is being listened for and opens a searchable list of every language the recognizer supports (served by GET /stt/languages with native names); a pick persists for that visitor. The reply language follows the question, not the chip.

Markup and events

::part(companion-mic), ::part(companion-stop), ::part(companion-dictation), ::part(language-chip) restyle the controls. Events: shukai:conversation:enter / :exit, shukai:voice:listening:start / :stop, shukai:voice:transcript, shukai:voice:speaking:start / :end, shukai:voice:error (Events). Shukai.speak(text), stopSpeaking(), startListening(), stopListening() (JavaScript API).

What a page may set

conversationModeEnabled and voiceInputEnabled can be overridden in ShukaiConfig, but the server refuses /tts with conversation mode off on the widget row and refuses /ws/stt with both switches off — a page cannot buy voice the owner did not turn on. voiceInputPace and voiceInputAutosend are served fixed; a page override changes only the widget’s own silence window and never the server’s end-of-utterance hold.

What it costs

A spoken answer costs the model’s credits plus a voice surcharge — one credit, three on a premium voice — added, never multiplied. The owner’s panel says which voice is premium. Credits and models.