Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
Diffusion Models & Image Generation focused on iterative generation under latency and quality trade-offs. Voice agents bring that pressure into live conversation: audio must stream, tools must resolve, and playback must stay interruptible.
A real-time voice AI agent has to listen, decide, speak, and recover quickly enough that the conversation still feels live. This design chapter covers speech pipelines, turn-taking, latency, tools, and fallback behavior.
An incident hotline feels broken if a user asks, "What is incident four two one nine?" and the voice on the other end pauses for two full seconds before answering. In those two seconds, the caller already wonders if the call dropped, or if the system heard the question at all. That silence kills the illusion that you're talking to something intelligent.
Human conversation runs on split-second timing. Studies of turn-taking show that people minimize silence and overlap across languages, with response timing varying by only hundreds of milliseconds across communities.[1] If a voice AI agent takes much longer than that, the conversation feels broken, no matter how smart the underlying model is. Building a system that listens, thinks, and speaks fast enough to feel natural is one of the hardest engineering problems in modern AI.
The concrete design is an incident-status assistant for an operations team. Trace one audio packet from the user's mouth to the model path and back to the speaker, then watch how the system recovers when the user interrupts mid-sentence. The same architecture choices apply to real-time voice products from IT help desks to emergency operations.
What makes voice agents harder than text chat?
Answer
Voice is continuous and time-sensitive. Silence, overlap, jitter, and interruptions are part of the interface, so the system must stream every stage and maintain state about what the user actually heard.
The latency illusion
The primary difference between a text-based chatbot and a voice agent is the continuous flow of time. A text interface lets the user wait patiently while a spinner animates. In a voice interface, silence is a signal. If the agent takes too long to reply, the user assumes it didn't hear them and will likely repeat themselves, causing confusion and overlapping speech.
A useful product target for an interactive support call is a measured time-to-first-audio (TTFA) budget around 500ms for the network and device slices you intend to support. Treat that as a service objective to validate, not a universal boundary: endpoint policy, tools, model choice, and network conditions move the distribution. Research systems such as AudioPaLM and SpeechGPT[2][3] show where speech-native models can go, while cascaded or hybrid pipelines remain useful when transcript inspection, debugging, and tool control matter.
Why might a team set a 500ms first-audio target instead of accepting a 2-second pause?
Answer
Human turn-taking expects quick feedback. A measured first-audio target near 500ms can preserve conversational flow on supported paths; a two-second silent tail is much more likely to be perceived as a missed turn or failed call.
A live routing line
Picture the problem before the architecture details. Building a voice agent is like running a live routing line where each component must hand off partial work without waiting for the full case to finish:
- Voice Activity Detection (VAD): Opens and closes the intake gate when speech starts and ends.
- Speech-to-Text (STT): Converts the incoming audio stream into partial text.
- LLM: Uses stable transcript segments to plan the response.
- Text-to-Speech (TTS): Starts speaking as soon as enough response text is available.
- Orchestrator: Coordinates handoffs, interruptions, and latency budgets.
If any stage drops packets or runs too slowly, the conversation breaks. The user experiences that as silence, overlap, or a response to a sentence they already corrected.
Why does every component need to hand off partial work?
Answer
Waiting for whole files, full transcripts, full LLM answers, or full audio synthesis adds seconds. Streaming handoffs let STT, LLM, and TTS overlap so first audio arrives quickly.
One packet through the incident-status agent
An engineer named Alex calls an incident hotline and says, "What is incident number four-two-one-nine?"
The audio passes through four steps before Alex hears a response:
- Listen (Voice Activity Detection): The system detects when Alex starts speaking and when Alex stops. VAD gates speech boundaries; a separate noise-suppression stage reduces noise mixed with speech.
- Transcribe (Speech-to-Text): The audio is converted to text. This must happen while Alex is still speaking (streaming), rather than waiting until Alex finishes.
- Think (Large Language Model): The model may prepare from stable partial transcripts, but the orchestrator must not speak a factual answer or execute a side effect until turn-commit policy accepts the input. Once committed, it streams the answer in speakable chunks.
- Speak (Text-to-Speech): The voice synthesis engine reads the first few words as soon as they arrive from the LLM, without waiting for the full sentence.
Each of these steps gets its own detailed look below. First, set the latency budget so you know what "fast enough" means in practice.
The latency budget: a worked example
Designing a voice AI agent requires balancing functional capabilities with strict latency constraints. The system needs to understand and generate natural language fast enough to mimic human turn-taking.
Functional requirements
- Real-time bidirectional conversation: Near-full-duplex experience where the user can interrupt at any time.
- Natural language understanding: Processing speech input into actionable intent.
- LLM reasoning: Generating context-aware responses and executing tools (incident lookup, severity summary).
- Natural speech synthesis: Generating audio with human-like prosody and emotion.
- Interruption handling (barge-in): The agent must stop speaking immediately when the user interrupts.
Example service objectives
- Speech-end to first audio: Track p50 and p95 from the acoustic end of user speech to first played assistant audio. This is the user-visible end-to-end clock; an operations line might start with a p95 target of 500ms on declared network slices.
- Commit to first audio: Track the internal clock from accepted turn commitment to first played audio. A route might start with a p95 target of 400ms. This excludes endpointing delay and makes post-commit LLM, TTS, and playout regressions easier to isolate.
- Mute on barge-in: Track user-speech onset to muted playback; a product might set a p95 target of 100ms.
- Concurrency: Size media and inference tiers from forecast active sessions and measured per-session work, rather than assuming
10k+. - Audio path: Negotiate supported codecs and sample rates, then measure jitter, loss, and intelligibility by device class.
- Availability: Choose an error budget for the call product and include media-connect, model, and tool failure modes.
Latency budget breakdown
Use 500ms speech-end TTFA and 400ms commit-to-audio as example targets, not laws of physics. A deployed route must report both distributions rather than promise one illustrative trace.
Trace the ranges from one shared event: acoustic speech end at 0ms. Streaming STT has already been consuming frames during the utterance, so only its finalization tail remains after speech ends. Read-only preparation may also be warm before commit. TTS starts only after an accepted turn yields a stable speakable clause.
Keep the two clocks separate. Speech-end TTFA includes endpointing and commit delay. Commit-to-audio starts at the accepted boundary and measures the internal response path. Streaming STT begins during speech rather than waiting for either clock.
Alex's "What is incident status?" query shows the timing path with real numbers:
| Stage | Position relative to acoustic speech end | What's happening |
|---|---|---|
| Streaming STT partials | Begin during speech, often before 0ms | Decoder emits "What is incident" while Alex is still talking |
| Turn commit | 0-60ms | Semantic end-of-turn (or hybrid policy) accepts that Alex's turn is complete; not a pure silence-threshold wait |
| STT finalization tail | 0-100ms | Final transcript catches up with audio already streamed during speech |
| First speakable LLM tokens / clause fragment | 60-240ms | Post-commit generation streams tokens; a speakable fragment can be ready before the full clause ends |
| First TTS chunk | 210-345ms | Synthesizer starts on the first speakable fragment (here at 210ms), while the clause may still finish at 240ms |
| Playout buffer | 345-380ms | Jitter buffer smooths network variance before the speaker |
If the post-speech spans ran one after another, their durations would add to about 510ms. In practice:
- STT started decoding before Alex finished the word "incident," so its full decoding cost isn't added after speech end.
- The orchestrator can pre-classify intent or start a cancellable read-only lookup from stable words; it can't commit an incident number, speak a result, or perform a side effect from an unstable transcript.
- After turn commitment, TTS starts synthesizing as soon as the LLM streams a speakable fragment (for example the start of "I found that incident"), before the full clause is finished. This overlap lets the TTS window open at 210ms while the LLM clause window still ends at 240ms.
- The
0-60mscommit window assumes semantic end-of-turn or an early-commit policy, not a fixed ~300ms silence threshold alone. Pure silence endpointing that waits ~300ms after speech end can't hit a 60ms commit; use the longer silence budget only when semantic EOT is unavailable or unreliable.
In the candidate trace below, first audio plays 380ms after acoustic speech end and 320ms after turn commit. The production decision depends on measured p50/p95 for both clocks, incorrect-commit rate, endpointing aggressiveness, model size, and network jitter.
1import json
2
3# One measured trace, in milliseconds after acoustic speech end.
4spans = {
5 "endpoint_commit": (0, 60),
6 "stt_finalization": (0, 100),
7 "llm_first_clause": (60, 240),
8 "tts_first_chunk": (210, 345),
9 "playout_buffer": (345, 380),
10}
11speech_end_ms = 0
12turn_commit_ms = spans["endpoint_commit"][1]
13first_audio_ms = spans["playout_buffer"][1]
14speech_end_target_ms = 500
15commit_target_ms = 400
16
17print(json.dumps({
18 "sequential_sum_ms": sum(end - start for start, end in spans.values()),
19 "speech_end_to_first_audio_ms": first_audio_ms - speech_end_ms,
20 "commit_to_first_audio_ms": first_audio_ms - turn_commit_ms,
21 "speech_end_target_ms": speech_end_target_ms,
22 "commit_target_ms": commit_target_ms,
23 "meets_speech_end_trace_target": first_audio_ms - speech_end_ms <= speech_end_target_ms,
24 "meets_commit_trace_target": first_audio_ms - turn_commit_ms <= commit_target_ms,
25 "ship_decision_requires_p95": True,
26}, indent=2))1{
2 "sequential_sum_ms": 510,
3 "speech_end_to_first_audio_ms": 380,
4 "commit_to_first_audio_ms": 320,
5 "speech_end_target_ms": 500,
6 "commit_target_ms": 400,
7 "meets_speech_end_trace_target": true,
8 "meets_commit_trace_target": true,
9 "ship_decision_requires_p95": true
10}Why is the sequential latency sum misleading?
Answer
Production voice pipelines overlap stages. STT starts during speech, reversible preparation can start from stable partials, and TTS can synthesize the first accepted clause before the full answer is done. Measure speech-end-to-audio for user experience and commit-to-audio for the internal response path.
How the pieces fit together
The architecture usually separates three concerns: client-side capture and playback, a latency-sensitive media edge, and the inference pipeline. Decoupling transport from inference matters because audio delivery has different failure modes from model execution. Packet loss, jitter, and echo cancellation must be handled on the media path even if the model stack is healthy.
In many deployments, a media gateway terminates WebRTC (Web Real-Time Communication) or SIP (Session Initiation Protocol), handles jitter buffers and routing, then forwards audio to inference over an internal low-latency link.[4] Not every system needs a full SFU (Selective Forwarding Unit) for a 1:1 assistant, but most serious deployments still benefit from a dedicated ingress tier close to the user.
Within inference, the voice pipeline orchestrates turn detection, STT or native-audio encoding, LLM reasoning, tool calls, and TTS or audio decoding. A conversation state manager sits alongside the pipeline, keeping track of dialogue turns, playback offsets, and tool results so the model only commits to what the user heard.
Follow the flow from left to right: capture, media edge, inference, and back to the speaker. Notice the barge-in path that instantly stops playback when the user interrupts.
Why split media transport from inference orchestration?
Answer
Audio transport handles jitter, packet loss, echo, routing, and playback timing. Inference handles STT, tools, LLM, and TTS. Their failure modes and scaling profiles differ.
Knowing when to listen: voice activity detection
Voice Activity Detection (VAD) determines when the user starts and stops speaking. In Alex's incident-status call, VAD is what tells the system, "Alex is done asking the question. You can start answering now."
VAD isn't the same thing as turn detection. VAD classifies each audio frame as speech or silence. Turn detection (endpointing) decides when the user's turn has finished so the agent can answer. Modern stacks increasingly layer three signals: VAD for raw speech presence, transcript-level endpointing for silence after stable words, and a model that judges semantic completeness from the partial transcript. Model-based end-of-turn detectors can commit a turn before trailing silence accumulates, cutting both clipped speakers and laggy responses.[5] You still keep a fast VAD underneath for instant barge-in.
Client-side VAD is common because it gives instant barge-in detection and avoids sending obvious silence. Server-side turn detection is also common because it centralizes tuning and can use richer context. This local wrapper represents a streaming VAD backend such as Silero VAD[6].
1from collections import deque
2from dataclasses import asdict, dataclass
3import json
4
5@dataclass
6class VADEvent:
7 type: str
8 frame: int
9 audio_ms: int
10
11class VoiceActivityDetector:
12 """Detect speech boundaries from one probability per 20ms audio frame."""
13
14 def __init__(
15 self,
16 frame_ms: int = 20,
17 speech_threshold: float = 0.5,
18 silence_duration_ms: int = 60,
19 prefix_padding_ms: int = 40,
20 ):
21 self.frame_ms = frame_ms
22 self.speech_threshold = speech_threshold
23 self.silence_frames_required = max(1, silence_duration_ms // frame_ms)
24 self.pre_roll = deque(maxlen=max(1, prefix_padding_ms // frame_ms))
25 self.speech_frames: list[int] = []
26 self.in_speech = False
27 self.trailing_silence = 0
28
29 def process_frame(self, frame_id: int, probability: float) -> VADEvent | None:
30 """Return speech boundary events for one audio frame."""
31 if probability >= self.speech_threshold:
32 if not self.in_speech:
33 self.in_speech = True
34 self.speech_frames = list(self.pre_roll)
35 self.speech_frames.append(frame_id)
36 self.trailing_silence = 0
37 return VADEvent("speech_start", frame_id, len(self.speech_frames) * self.frame_ms)
38 self.speech_frames.append(frame_id)
39 self.trailing_silence = 0
40 return None
41
42 if not self.in_speech:
43 self.pre_roll.append(frame_id)
44 return None
45
46 self.speech_frames.append(frame_id)
47 self.trailing_silence += 1
48 if self.trailing_silence < self.silence_frames_required:
49 return None
50
51 utterance_frames = self.speech_frames[:-self.trailing_silence]
52 self.in_speech = False
53 self.speech_frames = []
54 self.trailing_silence = 0
55 self.pre_roll.clear()
56 return VADEvent("speech_end", frame_id, len(utterance_frames) * self.frame_ms)
57
58probabilities = [0.04, 0.08, 0.66, 0.74, 0.69, 0.22, 0.15, 0.09]
59detector = VoiceActivityDetector()
60events = []
61
62for frame_id, probability in enumerate(probabilities):
63 event = detector.process_frame(frame_id, probability)
64 if event:
65 events.append(asdict(event))
66
67print(json.dumps(events, indent=2))1[
2 {
3 "type": "speech_start",
4 "frame": 2,
5 "audio_ms": 60
6 },
7 {
8 "type": "speech_end",
9 "frame": 7,
10 "audio_ms": 100
11 }
12]The demo uses a short silence_duration_ms = 60 so its output fits in eight frames. A phone support bot might begin testing around 300ms, then tune from measured false cutoffs and lag. Slow speakers may need longer thresholds; push-to-talk or command-style assistants can go shorter. A fixed silence threshold is the floor, not the ceiling: a semantic end-of-turn model can hold through mid-sentence pauses (a caller reciting a phone number digit by digit) and fire early once the utterance is complete, so you don't have to pick one silence value that hurts every other case.[5]
VAD deployment: client vs. server
Choosing where to run VAD changes responsiveness, bandwidth, and operational control. Client-side detection wins on local responsiveness. Server-side detection wins on centralized tuning and consistent turn boundaries across devices.
| Feature | Client-Side VAD | Server-Side Detection |
|---|---|---|
| Barge-in reaction | Can mute local playout without a network round trip | Includes transport delay before server action reaches playback |
| Bandwidth | Can suppress silence if upload policy uses its output | Receives the configured upstream stream continuously |
| Operational control | Requires testing across device/browser classes | Centralizes threshold and model tuning |
| Turn consistency | Device signals may differ | Can apply one server-side commit policy |
| Data exposure | Reduced only if capture/upload policy withholds silent frames | Depends on retained audio and server policy |
Many production systems use both: lightweight local VAD to mute playback instantly on barge-in, plus server-side turn detection to decide when transcripts are stable enough to trigger response generation.
The hybrid setup gives the user the fast interruption path they expect while keeping final turn boundaries consistent enough for transcripts, tools, and analytics.
Why run both local VAD and server-side turn detection?
Answer
Local VAD can mute local playback quickly and can suppress silence before upload when configured to do so. Server-side detection (silence endpointing plus a semantic end-of-turn model) gives centralized boundaries and richer context for deciding when the turn is truly complete, beyond the moment audio goes quiet.
Acoustic echo cancellation (AEC)
AEC prevents the agent from hearing itself. When the agent speaks and the user interrupts, the microphone picks up both the user's voice and the agent's own output playing from the speaker. Without AEC, the VAD would trigger on the agent's voice, creating an echo loop where the agent transcribes its own speech and responds to it.
Modern browser capture stacks expose echo-cancellation constraints so applications can request microphone input with system or remote audio removed before it reaches VAD and STT.[7] The browser or device owns that processing, and implementations vary. Treat the constraint as a request, then test the resulting audio path across speaker volume, microphone placement, rooms, and device classes.
One common failure is deploying voice agents without AEC testing in real acoustic environments. Laptop speakers and microphones placed close together, as in mobile phones, create strong echo that untested AEC configurations may fail to suppress. The symptom is brutal: the agent interrupts itself.
What happens if echo cancellation fails during agent playback?
Answer
The microphone can capture the agent's own voice, VAD treats it as user speech, STT transcribes it, and the agent can interrupt or respond to itself.
From sound to text: speech-to-text
Two primary approaches exist for real-time STT. In Alex's call, streaming STT is what lets the system start thinking about "What is incident" before Alex fully finishes saying "incident."
Streaming STT (recommended for latency)
Streaming STT sends partial transcripts as audio arrives, allowing the LLM to pre-read the turn and start cancellable, read-only preparation before the utterance is complete. Provider SDKs differ, but the state machine must separate unstable interim text from committed text that may enter history, trigger tools, or reach TTS.
1from dataclasses import asdict, dataclass
2import json
3
4@dataclass
5class TranscriptEvent:
6 kind: str
7 text: str
8 stable_prefix: str
9
10def common_prefix(left: str, right: str) -> str:
11 shared = []
12 for left_word, right_word in zip(left.split(), right.split()):
13 if left_word != right_word:
14 break
15 shared.append(left_word)
16 return " ".join(shared)
17
18def emit_transcript_events(partials: list[tuple[str, bool]]) -> list[TranscriptEvent]:
19 events = []
20 previous_text = ""
21
22 for text, is_final in partials:
23 stable_prefix = text if is_final else common_prefix(previous_text, text)
24
25 if is_final:
26 events.append(TranscriptEvent("final", text, stable_prefix))
27 else:
28 events.append(TranscriptEvent("interim", text, stable_prefix))
29 previous_text = text
30
31 return events
32
33partials = [
34 ("What is", False),
35 ("What is incident", False),
36 ("What is incident number four two", False),
37 ("What is incident number four two one nine?", True),
38]
39
40print(json.dumps([asdict(event) for event in emit_transcript_events(partials)], indent=2))1[
2 {
3 "kind": "interim",
4 "text": "What is",
5 "stable_prefix": ""
6 },
7 {
8 "kind": "interim",
9 "text": "What is incident",
10 "stable_prefix": "What is"
11 },
12 {
13 "kind": "interim",
14 "text": "What is incident number four two",
15 "stable_prefix": "What is incident"
16 },
17 {
18 "kind": "final",
19 "text": "What is incident number four two one nine?",
20 "stable_prefix": "What is incident number four two one nine?"
21 }
22]The shared-prefix heuristic is intentionally conservative: it only exposes words repeated across two successive hypotheses. Even that prefix is tentative because a provider can revise earlier words later. Use it only for reversible preparation. The commit boundary matters most when a partial transcript changes meaning. An incident lookup may be safe to prefetch and discard; closing an incident or promising mitigation status isn't.
1import json
2
3def permitted_actions(text: str, committed: bool) -> list[str]:
4 actions = ["classify_intent"]
5 if "incident" in text.lower():
6 actions.append("prefetch_read_only_status")
7 if committed:
8 actions.extend(["write_history", "speak_response"])
9 if "close" in text.lower():
10 actions.append("ask_for_close_confirmation")
11 return actions
12
13events = [
14 {"text": "Close incident four two one nine", "committed": False},
15 {"text": "Show incident four two one nine", "committed": True},
16]
17
18print(json.dumps([
19 {"committed": event["committed"], "actions": permitted_actions(**event)}
20 for event in events
21], indent=2))1[
2 {
3 "committed": false,
4 "actions": [
5 "classify_intent",
6 "prefetch_read_only_status"
7 ]
8 },
9 {
10 "committed": true,
11 "actions": [
12 "classify_intent",
13 "prefetch_read_only_status",
14 "write_history",
15 "speak_response"
16 ]
17 }
18]End-of-utterance STT (simpler, slightly higher latency)
Using a batch recognizer such as Whisper[8] after an endpoint event is much simpler, but it adds the endpoint wait and full decode after the user stops talking.
1import json
2
3def batch_stt_timeline(speech_ms: int, endpoint_ms: int, decode_ms: int) -> dict[str, int | str]:
4 acoustic_speech_end_ms = speech_ms
5 turn_commit_ms = acoustic_speech_end_ms + endpoint_ms
6 transcript_ready_ms = turn_commit_ms + decode_ms
7 return {
8 "approach": "end-of-utterance",
9 "speech_ms": speech_ms,
10 "endpoint_ms": endpoint_ms,
11 "decode_ms": decode_ms,
12 "acoustic_speech_end_ms": acoustic_speech_end_ms,
13 "turn_commit_ms": turn_commit_ms,
14 "first_transcript_ms": transcript_ready_ms,
15 }
16
17print(json.dumps(batch_stt_timeline(speech_ms=1200, endpoint_ms=300, decode_ms=450), indent=2))1{
2 "approach": "end-of-utterance",
3 "speech_ms": 1200,
4 "endpoint_ms": 300,
5 "decode_ms": 450,
6 "acoustic_speech_end_ms": 1200,
7 "turn_commit_ms": 1500,
8 "first_transcript_ms": 1950
9}| Approach | Latency | Complexity |
|---|---|---|
| Streaming STT | Illustrative first partial: ~50-150ms | High |
| End-of-utterance | Illustrative post-end decode: ~200-800ms+ | Low |
When is batch STT acceptable, and when is it a bad fit?
Answer
Batch STT is acceptable for non-live recordings or low-complexity prototypes. It's a bad fit for natural conversation because it waits until the user stops before decoding.
Thinking before the user finishes: LLM reasoning with streaming
After turn commitment, the LLM should emit a stable first speakable clause without waiting for a long response plan. Before commitment, the system may only do reversible preparation. This makes Time-to-First-Token (TTFT), the duration it takes for the LLM to produce its first output token, relevant without allowing an unstable transcript to become spoken truth; the serving mechanics are covered in Inference Mechanics.
Long inference-time reasoning and slow tools are poor candidates for a low-latency acknowledgement path unless measurements meet its TTFA objective. A voice agent can speak an honest acknowledgement quickly, run complex work asynchronously, and deliver the result when ready instead of pretending every question has an instant answer.
The helper below shows the voice-specific shape: short clauses, no Markdown, tool status stated early, and a stream of text chunks that TTS can consume before the full response is complete.
1import json
2
3def voice_chunks(transcript: str, incident_status: dict[str, str]) -> list[str]:
4 reply = (
5 f"Incident {incident_status['id']} is still open. "
6 f"Severity is {incident_status['severity']}. "
7 "I can text you the incident link."
8 )
9 clauses = [part.strip() + "." for part in reply.split(".") if part.strip()]
10 return clauses
11
12transcript = "What is incident number four two one nine?"
13chunks = voice_chunks(
14 transcript=transcript,
15 incident_status={"id": "4219", "severity": "sev-two"},
16)
17
18print(json.dumps({
19 "contains_incident": "incident" in transcript.lower(),
20 "max_words_per_clause": max(len(clause.split()) for clause in chunks),
21 "streamed_clauses": chunks,
22}, indent=2))1{
2 "contains_incident": true,
3 "max_words_per_clause": 7,
4 "streamed_clauses": [
5 "Incident 4219 is still open.",
6 "Severity is sev-two.",
7 "I can text you the incident link."
8 ]
9}Prompt engineering for voice is distinct from text. Voice prompts must explicitly forbid Markdown, which TTS reads poorly, and encourage front-loading the answer so meaningful audio is generated immediately.
Why should voice prompts forbid Markdown and long setup?
Answer
TTS reads markdown awkwardly, and long preambles delay useful audio. Voice responses should front-load the answer in short speakable clauses.
Serving optimizations that matter in voice
Live voice sessions stress inference differently from chat. TTFT matters more than long-form throughput because TTS can't speak until the first stable clause arrives. Session length also matters because a 20-minute call can accumulate enough context to make KV-cache management part of the latency budget, not a background infrastructure concern.
- Paged KV caches: Long-lived sessions fragment memory if every request expects one contiguous KV-cache region. PagedAttention-style allocators let serving systems grow and trim per-session context without constant copying, which helps keep latency stable across many simultaneous calls.[9]
- Speculative decoding: A draft model proposes tokens that the target model verifies in parallel, reducing decode latency when acceptance and implementation overhead are favorable.[10] It isn't an automatic TTFT improvement; benchmark TTFA and total clause latency on the served model pair.
- Aggressive context hygiene: Don't keep every interim transcript forever. Preserve final transcripts, tool results, and only the portion of assistant audio the user heard before barge-in.
Prompt caching is prefix economics
Prompt caching only helps when repeated requests preserve a matching prefix. Official OpenAI guidance recommends placing static instructions and examples at the beginning and variable content later.[11] Voice sessions are good candidates because system policy, voice style, and tool schemas often stay fixed across many turns.
Build the prompt in this order:
- Stable policy, output style, and safety instructions.
- Stable tool definitions and few-shot examples.
- Conversation history that grows by appending committed turns.
- Dynamic call state such as current incident result, timestamp, or playback acknowledgement.
Putting current_time=... or a rolling account balance near the top changes early tokens and can turn every later request into a cache miss. Keep volatile facts in a suffix or tool result, but never move instructions merely to chase hits when doing so changes model behavior or the authorization boundary.
Measure economics directly:
Provider eligibility, retention, and cached-input rates change, so read the usage metadata and current official pricing rather than assuming one discount. Track eligible prefix tokens, cached tokens, the cache-hit ratio, uncached input cost per completed call, and TTFA. A cache hit can lower repeated prefill work; it doesn't fix endpointing, tool, TTS, or playout latency.
Why should the current incident status appear after stable voice instructions and tool schemas?
Answer
The status changes often, so placing it near the top busts the reusable prefix. Keep the stable policy first and the dynamic state later, then verify the usage metadata, cost, latency, and unchanged behavior.
Why should interim transcripts not live forever in voice-agent context?
Answer
Interim text can be wrong and bloats long sessions. Keep final transcripts, tool results, and assistant audio actually heard; drop unstable partials and truncated speech.
Speaking without waiting for the full answer: TTS with streaming
Begin speaking before the full response is generated. The TTS engine receives a stream of text tokens, buffers them into sentences or phrases, and synthesizes audio chunks on the fly.
Waiting for the LLM to finish an entire sentence or paragraph before starting TTS can add seconds of latency. Stream tokens and synthesize clause-by-clause instead.
This function demonstrates buffering text into short speakable clauses. Waiting for a full paragraph is too slow, but firing on every token sounds choppy.
1import json
2
3def flush_speakable_clauses(tokens: list[str], max_chars: int = 52) -> list[str]:
4 buffer = ""
5 flushed = []
6
7 for token in tokens:
8 buffer += token
9 stripped = buffer.strip()
10 if stripped.endswith((",", ".", "!", "?")) or len(stripped) >= max_chars:
11 flushed.append(stripped)
12 buffer = ""
13
14 if buffer.strip():
15 flushed.append(buffer.strip())
16
17 return flushed
18
19tokens = [
20 "Incident ", "4219 ", "is ", "still ", "open, ",
21 "severity ", "is ", "sev-two. ",
22 "I ", "can ", "text ", "the ", "incident ", "link."
23]
24
25print(json.dumps(flush_speakable_clauses(tokens), indent=2))1[
2 "Incident 4219 is still open,",
3 "severity is sev-two.",
4 "I can text the incident link."
5]How parallel execution reduces latency
The end-to-end target starts at acoustic speech end, while the internal target starts at turn commit. The production pipeline overlaps stages rather than charging every component serially to either clock. A realistic overlap looks like this:
- VAD runs continuously, monitoring audio as it arrives and contributing a speech-end signal to endpointing.
- Streaming STT is already running during speech. Silence doesn't start recognition; it helps the endpoint policy decide when the current turn may commit.
- Reversible LLM preparation can start from stable STT partials, while spoken or side-effecting work waits for turn commitment.
- TTS begins synthesizing as soon as the committed response has a stable speakable clause, which may be well before the LLM finishes its full response.
- Audio playback begins as soon as the first TTS chunk is ready.
The illustrated trace lands around 300-450ms time to first audio (TTFA) on its assumed healthy path. Real TTFA depends on endpointing errors, model size, tool routing, and jitter-buffer depth; report its distribution for target network slices.
Overlap, not faster individual stages alone, drives the latency budget. VAD, STT, LLM, TTS, and playout each has latency, but the first audible word can arrive before every downstream job has fully completed.
Why synthesize clauses instead of individual tokens or full paragraphs?
Answer
Token-level TTS sounds choppy, while paragraph-level TTS adds seconds. Clause buffering gives natural phrasing while keeping first audio fast.
When the user interrupts: barge-in handling
When the user speaks over the agent ("barge-in"), the system must react instantly to maintain the illusion of a natural conversation. Imagine Alex asks about an incident, the agent starts explaining mitigation status, and Alex interrupts with "Wait, page the owner." The agent must stop immediately.
- Stop Playback: Immediately halt the audio output. A support product might begin with a p95 mute target below 100ms, then tune against measured paths.
- Cancel Generation: Abort in-flight TTS and LLM requests to save cost and compute.
- State Reconciliation: Update the conversation context to reflect only what the user heard.
Stopping the speaker is only half of barge-in. The next model call should remember only the part of the assistant response that crossed the speaker boundary.
The handler below shows conservative reconciliation at audio-chunk boundaries. When VAD detects fresh user speech during agent playback, the system must stop playout, cancel in-flight generation, and truncate assistant text to the portion confirmed through the speaker.
1from dataclasses import dataclass
2import json
3
4@dataclass
5class PlayedChunk:
6 text: str
7 text_end_char: int # Exclusive end offset in pending_text.
8 playback_end_ms: int
9
10class InterruptionHandler:
11 def __init__(self, pending_text: str, played_chunks: list[PlayedChunk]):
12 self.pending_text = pending_text
13 self.played_chunks = played_chunks
14
15 def reconcile(self, interrupt_ms: int) -> dict[str, object]:
16 heard_chars = self._chars_heard(interrupt_ms)
17 spoken_text = self.pending_text[:heard_chars].rstrip()
18 discarded_text = self.pending_text[heard_chars:].strip()
19 return {
20 "history_to_keep": [{"role": "assistant", "content": spoken_text}],
21 "discarded_generated_text": discarded_text,
22 }
23
24 def _chars_heard(self, interrupt_ms: int) -> int:
25 heard_chars = 0
26 for chunk in self.played_chunks:
27 if chunk.playback_end_ms <= interrupt_ms:
28 heard_chars = chunk.text_end_char
29 else:
30 break
31 return heard_chars
32
33pending = "Incident is open because the database failover is still running."
34chunks = [
35 PlayedChunk("Incident is", 11, 240),
36 PlayedChunk(" open", 16, 520),
37 PlayedChunk(" because the database", 37, 860),
38 PlayedChunk(" failover is still running.", 64, 1120),
39]
40
41result = InterruptionHandler(pending, chunks).reconcile(interrupt_ms=650)
42print(json.dumps(result, indent=2))1{
2 "history_to_keep": [
3 {
4 "role": "assistant",
5 "content": "Incident is open"
6 }
7 ],
8 "discarded_generated_text": "because the database failover is still running."
9}This tiny handler keeps only fully played chunks. text_end_char is an exclusive Python slice boundary: offset 16 keeps pending_text[:16], which is "Incident is open". In production, use smaller chunks or map synthesized spans to actual playout timestamps or RTP sequence ranges when you need finer reconciliation. Don't rely on model tokens alone. Jitter buffers can delay or drop chunks after synthesis, so "generated" isn't the same as "heard."
Why must barge-in history use "heard" text, not generated text?
Answer
The user may interrupt before playback finishes. If history includes unheard generated text, the model assumes shared context that the user never received, causing confusing follow-ups.
Cascaded pipeline vs. native audio
Design reviews often ask you to compare two design families: cascaded STT -> LLM -> TTS pipelines and native speech-to-speech models. Early research systems such as AudioPaLM and SpeechGPT showed the idea was viable.[2][3] Native audio is now a production API option: OpenAI's Realtime API supports live speech-to-speech sessions,[12] and Google documents Gemini Live native-audio sessions with voice activity detection and tool use.[13] Provider model catalogs change quickly, so verify current aliases when you implement the route. Native models reduce model handoffs and can retain prosodic signals that a text-only boundary loses, but they don't remove transport, VAD, AEC, or state-management problems.
Use the comparison as a control checklist. Native audio can shorten the model path, but you still need stateful sessions, transcript side channels, tool policy, and interruption handling.
Many commercial native-audio APIs use stateful sessions, not one-shot request/response calls. The client streams audio frames in, then receives incremental audio and transcript events back.
Session shape
1from collections import defaultdict
2import json
3
4events = [
5 {"type": "transcript_delta", "text": "Incident is"},
6 {"type": "audio_delta", "audio_ms": 160},
7 {"type": "transcript_delta", "text": " still open."},
8 {"type": "audio_delta", "audio_ms": 220},
9 {"type": "tool_call", "name": "send_incident_link"},
10]
11
12state = defaultdict(list)
13played_audio_ms = 0
14
15for event in events:
16 if event["type"] == "audio_delta":
17 played_audio_ms += event["audio_ms"]
18 elif event["type"] == "transcript_delta":
19 state["transcript"].append(event["text"])
20 elif event["type"] == "tool_call":
21 state["tool_calls"].append(event["name"])
22
23print(json.dumps({
24 "transcript_side_channel": "".join(state["transcript"]),
25 "played_audio_ms": played_audio_ms,
26 "tool_calls": state["tool_calls"],
27}, indent=2))1{
2 "transcript_side_channel": "Incident is still open.",
3 "played_audio_ms": 380,
4 "tool_calls": [
5 "send_incident_link"
6 ]
7}Audio tokenization
Published native-audio systems commonly avoid feeding raw PCM directly into a transformer. AudioPaLM and SpeechGPT use compressed speech representations rather than modeling every waveform sample as its own sequence element.[2][3]
Generation works in reverse: the model emits audio tokens or latents, a decoder reconstructs waveform chunks, and the client plays them with a jitter buffer. System-design point: compressed speech representations make sequence modeling practical while preserving tone, pacing, and other paralinguistic cues.
Trade-offs
Native audio can avoid some model handoffs and retain paralinguistic features such as tone, hesitation, and emphasis that a strict text bottleneck removes. Whether it produces faster or better interactions is an evaluation result, not an architectural guarantee.
However, native audio is harder to debug and harder to control. Without a clean intermediate transcript, it's tougher to inspect model failure modes, run deterministic tool policies, or align partially spoken output with conversation history. Modular pipelines are still easier to observe, test, and swap component-by-component.
When should you prefer a cascaded STT-LLM-TTS pipeline over native audio?
Answer
Prefer cascaded pipelines when observability, deterministic tool policy, component swaps, transcript inspection, or compliance logs matter more than maximum prosody and theoretical latency.
Moving audio across the internet
When moving live audio across the internet, transport choice sets baseline latency and jitter. For browser and mobile voice agents, WebRTC prefers a media-timed UDP path when available, but ICE/TURN deployments need relay and TCP/TLS fallback paths for restricted networks (RFC 8835). WebSockets (persistent, full-duplex TCP connections) remain useful for signaling, transcription-only streams, or server-to-server audio when you're willing to manage media buffering yourself.
TCP provides reliable, ordered byte delivery with sequence numbers, acknowledgments, and retransmission. That's great for text and file transfer, but it creates head-of-line blocking for live audio. If one segment is lost, later bytes wait behind recovery. In voice, that often shows up as stuttery playout or latency spikes.
On a WebRTC UDP media path, loss handling is deadline-aware: the stream can keep moving while packet-loss concealment fills short gaps. For conversation, concealing a short lost frame may be preferable to delaying later audio for recovery; verify this trade-off under the network slices you serve.
The deadline changes the protocol choice. RTP recovery helps only while a packet can still improve playout; TCP recovery must preserve byte order even if the voice moment has already passed.
Jitter buffers and packet loss concealment
Network jitter (variance in packet arrival time) is the enemy of smooth audio. A jitter buffer on the receiver side (media server or client) holds incoming packets for a short duration before playback. A team might begin experiments around 20-60ms, but the right value depends on network quality and how much delay your UX can tolerate.
- Static Jitter Buffer: Fixed size (e.g., 50ms). Simple but risks underruns if jitter exceeds buffer size.
- Adaptive Jitter Buffer: Dynamically resizes based on network conditions. Expands during congestion (adding latency) and shrinks during stability (reducing latency).
Browser WebRTC stacks already include adaptive jitter buffers and packet loss concealment (PLC). NetEQ is a well-known WebRTC receiver implementation of this idea.[14] When packets arrive late or disappear, the receiver can stretch buffered audio, extrapolate from recent audio, or ask the decoder to synthesize concealment frames instead of stalling playout.
Protocol comparison: WebRTC vs. WebSocket
| Feature | WebRTC Media (UDP Preferred, Fallbacks Possible) | WebSocket (TCP) |
|---|---|---|
| Transport | UDP preferred; relay or TCP/TLS fallback paths possible | TCP |
| Reliability Model | Media-timed (may use NACK/FEC, but playout stays time-bounded) | Byte-stream reliable (lost bytes block later bytes) |
| Latency Behavior | Low and stable when tuned | Can spike under loss |
| Congestion Control | Media-aware adaptation via RTP/RTCP feedback (implementation-dependent) | Managed by the TCP stack, not by media playout deadlines |
| Encryption | Mandatory (DTLS (Datagram Transport Layer Security) / SRTP (Secure Real-time Transport Protocol)) | TLS (over TCP) |
| Ideal Use Case | Real-time Voice/Video | Chat, Signaling, File Transfer |
WebRTC media is richer than "UDP means drop packets." On UDP media paths it layers RTP (Real-time Transport Protocol) / RTCP (RTP Control Protocol) feedback, jitter buffers, packet loss concealment (PLC), and sometimes NACK (Negative Acknowledgment) or FEC (Forward Error Correction).[4] ICE and TURN can select relay or fallback paths when direct UDP is unavailable (RFC 8835). The design goal is timeliness: recover loss when it can still help, otherwise keep audio moving.
Using WebSockets as a drop-in replacement for WebRTC in browser live audio creates hidden work. You can ship with WebSockets, but you must own chunking, playout buffering, backpressure, and worse behavior under packet loss.
Why does WebRTC usually beat WebSockets for browser live audio?
Answer
WebRTC uses media-timed transport with jitter buffers, RTP/RTCP feedback, packet loss concealment, and time-bounded playout. TCP WebSockets can stall later audio behind lost bytes.
Scaling to thousands of conversations
Designing voice AI for scale introduces challenges around state, media routing, and bursty GPU demand. For a system serving 10,000+ concurrent sessions, the ingress tier usually needs sticky routing or consistent hashing so packets for one conversation keep landing on the same media worker.
The media tier scales by active sessions and network state. Inference scales by model pressure, which is why separating these tiers usually beats one giant "voice server" process. The routing sketch uses a stable modulo hash only to show affinity with a fixed worker set. A production router should define what happens when workers join or fail, often with connection-aware routing or consistent hashing to limit remapping.
WebRTC connections carry session state. A specific media worker holds Datagram Transport Layer Security (DTLS) keys, Secure Real-time Transport Protocol (SRTP) state, jitter-buffer state, and often playback cursor. If a naive round-robin load balancer routes packets to a different worker mid-session, decryption and timing state break immediately.
Behind media workers, inference components should scale independently because their compute profiles differ. Turn detection and some STT stages often fit on CPU. Largest speech or LLM models usually need GPU. Independent scaling keeps GPU utilization high without overprovisioning lighter tiers.
1import hashlib
2import json
3
4workers = ["media-a", "media-b", "media-c"]
5
6def route(session_id: str) -> str:
7 digest = int(hashlib.sha256(session_id.encode()).hexdigest(), 16)
8 return workers[digest % len(workers)]
9
10session_id = "call-alex-4219"
11packet_routes = [route(session_id) for _ in range(4)]
12failed_worker = packet_routes[0]
13
14print(json.dumps({
15 "packet_routes": packet_routes,
16 "single_worker_during_session": len(set(packet_routes)) == 1,
17 "failed_worker": failed_worker,
18 "recovery": "reconnect_and_rebuild_session",
19}, indent=2))1{
2 "packet_routes": [
3 "media-c",
4 "media-c",
5 "media-c",
6 "media-c"
7 ],
8 "single_worker_during_session": true,
9 "failed_worker": "media-c",
10 "recovery": "reconnect_and_rebuild_session"
11}Why do WebRTC sessions need sticky routing?
Answer
A media worker holds DTLS keys, SRTP state, jitter buffers, and playback cursor. Moving packets to a different worker mid-session breaks decryption and timing state.
System design summary
Building a voice agent with a tight first-audio objective requires measuring every handoff across the processing pipeline. The architecture must be resilient to network jitter while handling the unpredictability of human conversation.
- Endpointing, STT, LLM, TTS, tools, and playout all consume or overlap within TTFA; slow work needs acknowledgement or asynchronous handling.
- STT streams partials, the LLM streams tokens, and TTS streams audio chunks. Native-audio routes still need measured incremental input, output, and interruption behavior.
- Interruption handling is a primary UX differentiator. Systems that don't stop speaking immediately feel robotic and rude.
- WebRTC is the live client-media default. Prefer its timed media path and plan for ICE/TURN fallback; WebSockets still fit signaling or managed server-side streaming.
- Native audio is a real option. Modular pipelines remain easier to debug and control, while native paths should be chosen only after measuring latency, prosody, policy, and state behavior.
- Prompt caching needs stable prefix ordering and usage-based verification. Dynamic timestamps, balances, and tool results belong after stable policy and schemas.
Diagram the lifecycle of a single voice packet from microphone to model and back, calculate the cumulative latency of a pipeline and identify the bottleneck, implement basic barge-in logic, and discuss the trade-offs between cascaded and native speech-to-speech models.