Real-Time Voice AI Agent

Diffusion Models & Image Generation focused on iterative generation under latency and quality trade-offs. Voice agents bring that pressure into live conversation: audio must stream, tools must resolve, and playback must stay interruptible.

A real-time voice AI agent has to listen, decide, speak, and recover quickly enough that the conversation still feels live. This design chapter covers speech pipelines, turn-taking, latency, tools, and fallback behavior.

Imagine calling customer support to track a delayed package. You say, "Where is my order?" and the voice on the other end pauses for two full seconds before answering. In those two seconds, you already wonder if the call dropped, or if the system heard you at all. That silence kills the illusion that you are talking to something intelligent.

Human conversation runs on split-second timing. Studies of turn-taking show that people minimize silence and overlap across languages, with response timing varying by only hundreds of milliseconds across communities.^[1] If a voice AI agent takes much longer than that, the conversation feels broken, no matter how smart the underlying model is. Building a system that listens, thinks, and speaks fast enough to feel natural is one of the hardest engineering problems in modern AI.

This article walks through the architecture of a production-grade voice agent, using a concrete example: a package-tracking assistant for an online store. You will follow a single audio packet from the user's mouth to the model's "brain" and back to the speaker, learn why every millisecond matters, and see how the system recovers when the user interrupts mid-sentence. The concepts here apply to any real-time voice product, from customer support to warehouse coordination.

The latency illusion

The primary difference between a text-based chatbot and a voice agent is the continuous flow of time. A text interface lets the user wait patiently while a spinner animates. In a voice interface, silence is a signal. If the agent takes too long to reply, the user assumes it didn't hear them and will likely repeat themselves, causing confusion and overlapping speech.

A useful product target for an interactive support call is a measured time-to-first-audio (TTFA) budget around 500ms for the network and device slices you intend to support. Treat that as a service objective to validate, not a universal boundary: endpoint policy, tools, model choice, and network conditions move the distribution. Research systems such as AudioPaLM and SpeechGPT^[2][3] show where speech-native models can go, while cascaded or hybrid pipelines remain useful when transcript inspection, debugging, and tool control matter.

A live routing line

Before analyzing the architecture, it helps to picture the problem. Building a voice agent is like running a live routing line where each component must hand off partial work without waiting for the full case to finish:

• Voice Activity Detection (VAD): Opens and closes the intake gate when speech starts and ends.
• Speech-to-Text (STT): Converts the incoming audio stream into partial text.
• LLM: Uses stable transcript segments to plan the response.
• Text-to-Speech (TTS): Starts speaking as soon as enough response text is available.
• Orchestrator: Coordinates handoffs, interruptions, and latency budgets.

If any stage drops packets or runs too slowly, the conversation breaks. The user experiences that as silence, overlap, or a response to a sentence they already corrected.

One packet's journey: the package-tracking agent

To make this concrete, imagine a customer named Alex who calls a voice support line for an online store. Alex says, "Where is my order number four-two-one-nine?"

Here is what happens to that audio, step by step, before Alex hears a response:

1. Listen (Voice Activity Detection): The system detects when Alex starts speaking and, most importantly, when Alex stops. It keeps background noise out of the speech stream.
2. Transcribe (Speech-to-Text): The audio is converted to text. This must happen while Alex is still speaking (streaming), not just after Alex finishes.
3. Think (Large Language Model): The model may prepare from stable partial transcripts, but the orchestrator must not speak a factual answer or execute a side effect until turn-commit policy accepts the input. Once committed, it streams the answer in speakable chunks.
4. Speak (Text-to-Speech): The voice synthesis engine reads the first few words as soon as they arrive from the LLM, without waiting for the full sentence.

Each of these steps gets its own detailed look below. First, let's set the latency budget so you know what "fast enough" means in practice.

The latency budget: a worked example

Designing a voice AI agent requires balancing functional capabilities with strict latency constraints. The system needs to understand and generate natural language fast enough to mimic human turn-taking.

Functional requirements

• Real-time bidirectional conversation: Near-full-duplex experience where the user can interrupt at any time.
• Natural language understanding: Processing speech input into actionable intent.
• LLM reasoning: Generating context-aware responses and executing tools (order lookup, delivery estimation).
• Natural speech synthesis: Generating audio with human-like prosody and emotion.
• Interruption handling (barge-in): The agent must stop speaking immediately when the user interrupts.

Example service objectives

• Time to first audio: Track p50 and p95 from committed turn end to first played audio; a support line might start with a p95 target of 500ms on declared network slices.
• Mute on barge-in: Track user-speech onset to muted playback; a product might set a p95 target of 100ms.
• Concurrency: Size media and inference tiers from forecast active sessions and measured per-session work, rather than assuming 10k+.
• Audio path: Negotiate supported codecs and sample rates, then measure jitter, loss, and intelligibility by device class.
• Availability: Choose an error budget for the call product and include media-connect, model, and tool failure modes.

Latency budget breakdown

Use 500ms as an example target, not a law of physics. A deployed route must report TTFA distributions rather than promise one illustrative trace.

Trace the ranges: each stage adds time, but streaming overlap allows read-only preparation before the final transcript and lets TTS start once an accepted turn has yielded a stable speakable clause.

Treat the 500ms number as a sequential accounting exercise. Good implementations hide some of that with streaming STT, streaming LLM output, and early TTS.

Let's walk through Alex's "Where is my order?" query with real numbers:

If these ran one after another, the total would be about 500ms. But in practice:

• STT starts decoding while Alex is still finishing the word "order."
• The orchestrator can pre-classify intent or start a cancellable read-only lookup from stable words; it cannot commit an order number, speak a result, or perform a side effect from an unstable transcript.
• After turn commitment, TTS starts synthesizing as soon as the LLM produces a stable clause like "I found that order."

In one candidate trace, this overlap may deliver first audio in 300-450ms. The production decision depends on measured p50/p95 TTFA, incorrect-commit rate, endpointing aggressiveness, model size, and network jitter.

How the pieces fit together

The architecture usually separates three concerns: client-side capture and playback, a latency-sensitive media edge, and the inference pipeline. Decoupling transport from inference matters because audio delivery has different failure modes from model execution. Packet loss, jitter, and echo cancellation must be handled on the media path even if the model stack is healthy.

In many deployments, a media gateway terminates WebRTC (Web Real-Time Communication) or SIP (Session Initiation Protocol), handles jitter buffers and routing, then forwards audio to inference over an internal low-latency link.^[4] Not every system needs a full SFU (Selective Forwarding Unit) for a 1:1 assistant, but most serious deployments still benefit from a dedicated ingress tier close to the user.

Within inference, the voice pipeline orchestrates turn detection, STT or native-audio encoding, LLM reasoning, tool calls, and TTS or audio decoding. A conversation state manager sits alongside the pipeline, keeping track of dialogue turns, playback offsets, and tool results so the model only commits to what the user actually heard.

Follow the flow from left to right: capture, media edge, inference, and back to the speaker. Notice the barge-in path that instantly stops playback when the user interrupts.

Knowing when to listen: voice activity detection

Voice Activity Detection (VAD) determines when the user starts and stops speaking. In Alex's package-tracking call, VAD is what tells the system, "Alex is done asking the question. You can start answering now."

VAD is not the same thing as turn detection. VAD classifies each audio frame as speech or silence. Turn detection (endpointing) decides when the user's turn is actually finished so the agent can answer. Modern stacks increasingly layer three signals: VAD for raw speech presence, transcript-level endpointing for silence after stable words, and a model that judges semantic completeness from the partial transcript. Model-based end-of-turn detectors can commit a turn before trailing silence accumulates, cutting both clipped speakers and laggy responses.^[5] You still keep a fast VAD underneath for instant barge-in.

Client-side VAD is common because it gives instant barge-in detection and avoids sending obvious silence. Server-side turn detection is also common because it centralizes tuning and can use richer context. The implementation below shows a generic local wrapper around a streaming VAD backend such as Silero VAD^[6].

silence_duration_ms = 300 is a good starting point, not a universal truth. Phone support bots often go longer to avoid clipping slow speakers. Push-to-talk or assistant experiences can go shorter. A fixed silence threshold is the floor, not the ceiling: a semantic end-of-turn model can hold through mid-sentence pauses (a caller reciting a phone number digit by digit) and fire early once the utterance is clearly complete, so you do not have to pick one silence value that hurts every other case.^[5]

VAD deployment: client vs. server

Choosing where to run VAD changes responsiveness, bandwidth, and operational control. Client-side detection wins on local responsiveness. Server-side detection wins on centralized tuning and consistent turn boundaries across devices.

Many production systems use both: lightweight local VAD to mute playback instantly on barge-in, plus server-side turn detection to decide when transcripts are stable enough to trigger response generation.

The hybrid setup gives the user the fast interruption path they expect while keeping final turn boundaries consistent enough for transcripts, tools, and analytics.

Acoustic echo cancellation (AEC)

A critical but often overlooked component is AEC, which prevents the agent from hearing itself. When the agent speaks and the user interrupts, the microphone picks up both the user's voice and the agent's own output playing from the speaker. Without AEC, the VAD would trigger on the agent's voice, creating an echo loop where the agent transcribes its own speech and responds to it.

Modern browser capture stacks expose echo-cancellation constraints so applications can request microphone input with system or remote audio removed before it reaches VAD and STT.^[7] This happens in the time domain using adaptive filters that model the room's acoustic characteristics. The filter continuously adjusts to account for speaker placement, room reverberation, and device variations.

One common failure is deploying voice agents without AEC testing in real acoustic environments. Laptop speakers and microphones placed close together, as in mobile phones, create strong echo that untested AEC configurations may fail to suppress. The symptom is brutal: the agent interrupts itself.

From sound to text: speech-to-text

Two primary approaches exist for real-time STT. In Alex's call, streaming STT is what lets the system start thinking about "Where is my order" before Alex fully finishes saying "order."

Streaming STT (recommended for latency)

Streaming STT sends partial transcripts as audio arrives, allowing the LLM to pre-read the turn and start cancellable, read-only preparation before the utterance is complete. Provider SDKs differ, but the state machine must separate unstable interim text from committed text that may enter history, trigger tools, or reach TTS.

The commit boundary matters most when a partial transcript changes meaning. A package lookup may be safe to prefetch and discard; canceling an order or promising an ETA is not.

End-of-utterance STT (simpler, slightly higher latency)

Using a batch recognizer such as Whisper^[8] after speech_end is much simpler, but it adds an extra decode step after the user stops talking.

Thinking before the user finishes: LLM reasoning with streaming

After turn commitment, the LLM should emit a stable first speakable clause without waiting for a long response plan. Before commitment, the system may only do reversible preparation. This makes Time-to-First-Token (TTFT), the duration it takes for the LLM to produce its first output token, relevant without allowing an unstable transcript to become spoken truth; the serving mechanics are covered in Inference Mechanics.

Long inference-time reasoning and slow tools are poor candidates for a low-latency acknowledgement path unless measurements meet its TTFA objective. A voice agent can speak an honest acknowledgement quickly, run complex work asynchronously, and deliver the result when ready instead of pretending every question has an instant answer.

The helper below shows the voice-specific shape: short clauses, no Markdown, tool status stated early, and a stream of text chunks that TTS can consume before the full response is complete.

Prompt engineering for voice is distinct from text. Voice prompts must explicitly forbid Markdown, which TTS reads poorly, and encourage front-loading the answer so meaningful audio is generated immediately.

Serving optimizations that matter in voice

Live voice sessions stress inference differently from chat. TTFT matters more than long-form throughput because TTS can't speak until the first stable clause arrives. Session length also matters because a 20-minute call can accumulate enough context to make KV-cache management part of the latency budget, not just an infrastructure detail.

• Paged KV caches: Long-lived sessions fragment memory if every request expects one contiguous KV region. PagedAttention-style allocators let serving systems grow and trim per-session context without constant copying, which helps keep latency stable across many simultaneous calls.^[9]
• Speculative decoding: A draft model proposes tokens that the target model verifies in parallel, reducing decode latency when acceptance and implementation overhead are favorable.^[10] It is not an automatic TTFT improvement; benchmark TTFA and total clause latency on the served model pair.
• Aggressive context hygiene: Don't keep every interim transcript forever. Preserve final transcripts, tool results, and only the portion of assistant audio the user actually heard before barge-in.

Speaking without waiting for the full answer: TTS with streaming

The key optimization is to begin speaking before the full response is generated. The TTS engine receives a stream of text tokens, buffers them into sentences or phrases, and synthesizes audio chunks on the fly.

Waiting for the LLM to finish an entire sentence or paragraph before starting TTS can add seconds of latency. Stream tokens and synthesize clause-by-clause instead.

The function below demonstrates buffering text into short speakable clauses. Waiting for a full paragraph is too slow, but firing on every token sounds choppy.

How parallel execution reduces latency

The 500ms budget represents the theoretical sequential sum of all components, but the production pipeline overlaps stages to fit well under that target. Here is a realistic overlap:

• VAD runs continuously, monitoring audio as it arrives. It triggers the STT call the moment silence is detected.
• STT runs in parallel with the user still finishing their last words. The first partial transcript arrives while speech is ongoing.
• Reversible LLM preparation can start from stable STT partials, while spoken or side-effecting work waits for turn commitment.
• TTS begins synthesizing as soon as the committed response has a stable speakable clause, which may be well before the LLM finishes its full response.
• Audio playback begins as soon as the first TTS chunk is ready.

The illustrated trace lands around 300-450ms time to first audio (TTFA) on its assumed healthy path. Real TTFA depends on endpointing errors, model size, tool routing, and jitter-buffer depth; report its distribution for target network slices.

The key idea is overlap, not magic speed. VAD, STT, LLM, TTS, and playout each has latency, but the first audible word can arrive before every downstream job has fully completed.

When the user interrupts: barge-in handling

When the user speaks over the agent ("barge-in"), the system must react instantly to maintain the illusion of a natural conversation. Imagine Alex asks about an order, the agent starts explaining the return policy, and Alex interrupts with "Actually, just cancel it." The agent must stop immediately.

1. Stop Playback: Immediately halt the audio output (less than 100ms).
2. Cancel Generation: Abort in-flight TTS and LLM requests to save cost and compute.
3. State Reconciliation: Update the conversation context to reflect only what the user actually heard.

The hard part isn't stopping the speaker. The hard part is making the next model call remember only the part of the assistant response that crossed the speaker boundary.

The handler below shows the core reconciliation logic. When VAD detects fresh user speech during agent playback, the system must stop playout, cancel in-flight generation, and truncate assistant text to the portion that actually reached the speaker.

In production, map synthesized chunks to actual playout timestamps or RTP sequence ranges, not to model tokens alone. Jitter buffers can delay or drop chunks after synthesis, so "generated" is not the same as "heard."

Cascaded pipeline vs. native audio

Design reviews often ask you to compare two design families: cascaded STT -> LLM -> TTS pipelines and native speech-to-speech models. Early research systems such as AudioPaLM and SpeechGPT showed the idea was viable.^[2][3] As of May 28, 2026, native audio is a production API option: OpenAI documents gpt-realtime for realtime audio input and output,^[11] and Google documents Gemini Live native-audio sessions with voice activity detection and tool use.^[12] Native models reduce model handoffs and can retain prosodic signals that a text-only boundary loses, but they do not remove transport, VAD, AEC, or state-management problems.

Use the comparison as a control checklist. Native audio can shorten the model path, but you still need stateful sessions, transcript side channels, tool policy, and interruption handling.

Many commercial native-audio APIs use stateful sessions, not one-shot request/response calls. The client streams audio frames in, then receives incremental audio and transcript events back.

Session shape

Audio tokenization

Many native audio systems bypass feeding raw PCM directly into the transformer. They first compress audio into acoustic tokens or other learned latent representations, then run the sequence model over those shorter units.

Generation works in reverse: the model emits audio tokens or latents, a decoder reconstructs waveform chunks, and the client plays them with a jitter buffer. System-design point: compressed speech representations make sequence modeling practical while preserving tone, pacing, and other paralinguistic cues.

Trade-offs

Native audio can avoid some model handoffs and retain paralinguistic features such as tone, hesitation, and emphasis that a strict text bottleneck removes. Whether it produces faster or better interactions is an evaluation result, not an architectural guarantee.

However, native audio is harder to debug and harder to control. Without a clean intermediate transcript, it's tougher to inspect model failure modes, run deterministic tool policies, or align partially spoken output with conversation history. Modular pipelines are still easier to observe, test, and swap component-by-component.

Moving audio across the internet

When moving live audio across the internet, transport choice sets baseline latency and jitter. For browser and mobile voice agents, WebRTC prefers a media-timed UDP path when available, but ICE/TURN deployments need relay and TCP/TLS fallback paths for restricted networks.^[4] WebSockets (persistent, full-duplex TCP connections) remain useful for signaling, transcription-only streams, or server-to-server audio when you are willing to manage media buffering yourself.

TCP guarantees delivery by acknowledging every packet. That's great for text and file transfer, but it creates head-of-line blocking for live audio. If one packet is lost, later packets wait behind the retransmission. In voice, that often shows up as stuttery playout or latency spikes in the hundreds of milliseconds.

On a WebRTC UDP media path, loss handling is deadline-aware: the stream can keep moving while packet-loss concealment fills short gaps. For conversation, concealing a short lost frame may be preferable to delaying later audio for recovery; verify this trade-off under the network slices you serve.

The difference is a deadline. RTP recovery helps only while a packet can still improve playout; TCP recovery must preserve byte order even if the voice moment has already passed.

Jitter buffers and packet loss concealment

Network jitter (variance in packet arrival time) is the enemy of smooth audio. A jitter buffer on the receiver side (media server or client) holds incoming packets for a short duration before playback. Around 20-60ms is a common starting range, but the right value depends on network quality and how much delay your UX can tolerate.

• Static Jitter Buffer: Fixed size (e.g., 50ms). Simple but risks underruns if jitter exceeds buffer size.
• Adaptive Jitter Buffer: Dynamically resizes based on network conditions. Expands during congestion (adding latency) and shrinks during stability (reducing latency).

Browser WebRTC stacks already include adaptive jitter buffers and packet loss concealment (PLC). NetEQ is a well-known WebRTC receiver implementation of this idea.^[13] When packets arrive late or disappear, the receiver can stretch buffered audio, extrapolate from recent audio, or ask the decoder to synthesize concealment frames instead of stalling playout.

Protocol comparison: WebRTC vs. WebSocket

WebRTC media is richer than "UDP means drop packets." On UDP media paths it layers RTP (Real-time Transport Protocol) / RTCP (RTP Control Protocol) feedback, jitter buffers, packet loss concealment (PLC), and sometimes NACK (Negative Acknowledgment) or FEC (Forward Error Correction). ICE and TURN can select relay or fallback paths when direct UDP is unavailable.^[4] The design goal is timeliness: recover loss when it can still help, otherwise keep audio moving.

Using WebSockets as a drop-in replacement for WebRTC in browser live audio creates hidden work. You can ship with WebSockets, but you must own chunking, playout buffering, backpressure, and worse behavior under packet loss.

Scaling to thousands of conversations

Designing voice AI for scale introduces challenges around state, media routing, and bursty GPU demand. For a system serving 10,000+ concurrent sessions, the ingress tier usually needs sticky routing or consistent hashing so packets for one conversation keep landing on the same media worker.

The media tier scales by active sessions and network state. The inference tier scales by model pressure, which is why separating these tiers usually beats one giant "voice server" process.

WebRTC connections are deeply stateful. A specific media worker holds Datagram Transport Layer Security (DTLS) keys, Secure Real-time Transport Protocol (SRTP) state, jitter-buffer state, and often playback cursor. If a naive round-robin load balancer routes packets to a different worker mid-session, decryption and timing state break immediately.

Behind media workers, inference components should scale independently because their compute profiles differ. Turn detection and some STT stages often fit on CPU. Largest speech or LLM models usually need GPU. Independent scaling keeps GPU utilization high without overprovisioning lighter tiers.

Mastery check

At this point you should be able to defend the full live loop: how first audio arrives fast, how barge-in trims history to heard audio, and why media routing must stay stateful even when model pools scale independently.

Common pitfalls

These are the failure patterns that most often break the voice illusion. In a design review, name the symptom, cause, and fix instead of blaming the model generically.

Symptom: The agent pauses for one or two seconds before it even starts thinking. Cause: You batch-transcribed the whole utterance instead of using streaming STT partials. Fix: Emit stable early text so the LLM can pre-read the turn before the final transcript lands.

Symptom: The answer is correct, but the long silence before speech makes the system feel dead. Cause: You waited for full LLM completion before sending anything to TTS. Fix: Buffer text into short speakable clauses and flush them as soon as they are stable enough to say aloud.

Symptom: The agent speaks an incorrect order number or starts an action the caller immediately corrects. Cause: You treated an unstable STT partial as a committed turn. Fix: Limit partial-transcript work to reversible preparation; authorize speech and side effects only after turn commitment and required confirmation.

Symptom: Slow speakers get clipped, or every caller feels like they are talking to a laggy robot. Cause: Endpointing is tuned backward for the call type. Fix: Tune silence thresholds and semantic end-of-turn rules against false cutoffs, barge-in latency, and time-to-first-audio.

Symptom: After an interruption, the agent answers as if the user heard words that never played. Cause: You stored generated text instead of playback-confirmed text. Fix: Align synthesized chunks to playout timestamps or RTP ranges and keep only the heard prefix in history.

Symptom: Browser audio works in demos but stalls badly when the network drops packets. Cause: You treated WebSockets as a drop-in replacement for WebRTC media. Fix: Prefer WebRTC for live browser audio, or explicitly own buffering, backpressure, and loss behavior yourself.

Symptom: The agent sounds clever after several seconds, but conversation feels broken. Cause: You put deep reasoning or heavy tool latency directly in the live voice loop. Fix: Keep the live model fast and push expensive reasoning into async tools, follow-up workflows, or background actions.

Follow-up questions

The questions below belong in the article, not in metadata, because they are part of the lesson. A reader should be able to answer them from the architecture they just built up.

Evaluation rubric

• Foundational: Explain an example 500ms first-audio objective across endpointing, STT, LLM, TTS, and playout, then name what must be measured before shipping.
• Intermediate: Draw the streaming path from mic capture to speaker playout and identify where stable partials allow overlap.
• Intermediate: Show how barge-in cancels work, trims history to heard audio, and avoids replaying unheard text into the next turn.
• Advanced: Defend WebRTC over WebSockets for browser live audio using deadline-based playout, jitter buffers, and head-of-line blocking.
• Advanced: Choose between cascaded and native audio architectures based on observability, tool control, prosody, and failure recovery.
• Advanced: Justify session scaling with sticky media routing and independently scaled inference pools.

What to build next

The best way to internalize this architecture is to build something small and measurable. Here are three progressive projects you can add to a portfolio:

1. Voice Mirror: A simple agent that transcribes what you say and speaks it back in a different voice. This teaches you how to wire VAD, STT, and TTS together without the complexity of LLM reasoning.

2. Smart Store Controller: A voice agent that can look up mock order status and initiate returns via function calls. This adds tool use and teaches you how latency changes when the LLM must call an API before responding.

3. Roleplay Support Agent: A voice agent that uses a clause buffer and records TTFA distributions while giving multi-sentence responses about delivery exceptions or refund policies. This teaches you to tune silence_duration_ms, clause flushing, and barge-in handling against a declared objective.

Key takeaways and next step

Building a voice agent with a tight first-audio objective requires measuring every handoff across the processing pipeline. The architecture must be resilient to network jitter while handling the unpredictability of human conversation.

• Measured latency objective: Endpointing, STT, LLM, TTS, tools, and playout all consume or overlap within TTFA; slow work needs acknowledgement or asynchronous handling.
• Stream everything: STT streams partials, the LLM streams tokens, and TTS streams audio chunks.
• Interruption handling: This is a primary UX differentiator. Systems that don't stop speaking immediately feel robotic and rude.
• WebRTC is the live client-media default: Prefer its timed media path and plan for ICE/TURN fallback; WebSockets still fit signaling or managed server-side streaming.
• Native audio is a real option: Modular pipelines remain easier to debug and control, while native paths should be chosen only after measuring latency, prosody, policy, and state behavior.

You should now be able to diagram the lifecycle of a single voice packet from microphone to model and back, calculate the cumulative latency of a pipeline and identify the bottleneck, implement basic barge-in logic, and discuss the trade-offs between cascaded and native speech-to-speech models.