Real-Time Speech Translation Architecture: Gemini 3.5 Live + Web Audio API (Tolky Case Study)
How Tolky streams spoken translations in the browser with sub-second latency — the Web Audio capture pipeline, the Gemini 3.5 Live transport, and the trade-offs that made it production-grade.
Tolky is a browser-based real-time speech translation product — speak Dutch, your counterpart hears Turkish, instantly, with no app install. The hard part is not the translation; it is doing it with low enough latency that a conversation still feels like a conversation. This is the architecture.
Capture: Web Audio API
Everything starts with getUserMedia and an AudioContext. We pull the mic into a ScriptProcessor-style worklet and chunk frames small enough to stream (tens of milliseconds). The constraints here are real: too large a chunk and latency piles up; too small and you spend the budget on framing overhead. The sweet spot for conversational speech landed around a specific frame size we tuned empirically.
Transport: streaming to Gemini 3.5 Live
Gemini 3.5 Live accepts a bidirectional audio stream over WebSocket. We push frames up and receive translated audio + transcript back down the same socket. The key engineering decision was keeping the socket warm and never re-negotiating mid-utterance — re-handshakes are the single biggest source of perceived "lag spikes."
Playback without crosstalk
The translated audio has to play out without feeding back into the capture loop. We split input and output gain stages explicitly and gate capture during playback windows. Get this wrong and the model happily translates its own output into an infinite loop.
Why browser-native matters
A doctor with limited Dutch and a patient who speaks only Turkish cannot both install an app, make accounts, and pair devices in the waiting room. Browser-native means one person opens a link. That single product constraint shaped every architectural choice: Web Audio (not a native SDK), WebSocket (not a heavier transport), no install (not a mobile app). GDPR-conform EU hosting and a signed DPA closed the compliance loop for the B2B tier — municipalities, GP practices, and schools will not sign anything less.