Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
Alex calls the incident hotline and asks, "What is incident four two one nine?" At two seconds, the on-call dashboard still shows a connected session and incoming frames, but no answer has reached the speaker. Alex wonders whether the call dropped. A text chatbot can hide behind a spinner. A voice agent can't: silence is a signal.
The previous chapter treated generation as a latency-versus-quality budget you can spend offline. This one spends that budget while Alex is waiting on the line. Audio has to stream, tools have to resolve without blocking the first word, and playback has to stop the moment Alex interrupts.
Human turn-taking is unforgiving. Across ten languages, Stivers et al. found that people minimize both silence and overlap. The cross-language mean gap was 208ms, and language-specific means stayed within about 250ms of that mean.[1] A 500ms first-audio target is already slower than that human mean. Treat it as a product SLO you measure on the network slices you support, not as a claim that the call will feel like two people in a room.
What makes voice agents harder than text chat?
Answer
Voice is continuous and time-sensitive. Silence, overlap, jitter, and interruptions are part of the interface, so the system must stream every stage and maintain state about what the user actually heard.
The latency illusion
If the agent takes too long to reply, Alex assumes it didn't hear the question and repeats it. That second attempt overlaps the late first answer, and the call unravels.
Before picking a model, name the clock you need to win. If acoustic speech ends at 0ms but endpointing commits at 300ms, no decoder can recover those 300ms. If first audio is late after a quick commit, look at generation, synthesis, transport, and playout instead.
A useful product target for this incident line is a measured time-to-first-audio (TTFA) budget around 500ms on the network and device slices you intend to support. Endpoint policy, tools, model choice, and jitter move the distribution, so ship from p50/p95, not from one lucky trace. Research systems such as AudioPaLM and SpeechGPT showed that speech-native models can collapse some of the speech-recognition, language-model, and speech-synthesis hops,[2][3] but cascaded pipelines remain the right default when you need inspectable transcripts, vendor swaps, or deterministic tool gates.
Why might a team set a 500ms first-audio target instead of accepting a 2-second pause?
Answer
Human turn-taking expects quick feedback. A measured first-audio target near 500ms can preserve conversational flow on supported paths; a two-second silent tail is much more likely to be perceived as a missed turn or failed call.
Five stages that must overlap
Each stage has to hand off partial work. Waiting for a full recording, a full transcript, a full LLM answer, or a fully synthesized paragraph adds seconds Alex won't sit through.
- Voice Activity Detection (VAD): Marks when speech starts and ends in each audio frame.
- Speech-to-Text (STT): Converts the incoming stream into partial text while Alex is still talking.
- LLM: Uses stable transcript segments to prepare, then speaks only after turn-commit policy accepts the input.
- Text-to-Speech (TTS): Starts speaking as soon as a first clause is stable.
- Orchestrator: Coordinates handoffs, interruptions, tool side effects, and both latency clocks.
Those stages are a schedule, not a checklist. If STT waits for a recording, LLM waits for a final transcript, or TTS waits for a complete answer, each wait lands on Alex's silence clock. If one stage drops packets or runs too slowly, Alex hears silence, overlap, or an answer to a sentence already corrected.
Why does every component need to hand off partial work?
Answer
Waiting for whole files, full transcripts, full LLM answers, or full audio synthesis adds seconds. Streaming handoffs let STT, LLM, and TTS overlap so first audio arrives quickly.
One packet through the incident-status agent
Follow Alex's utterance through those handoffs and ask where the first audible clause can start. Then put numbers on both clocks and let an interruption test whether state reflects what Alex heard.
The audio passes through four steps before Alex hears a response:
- Listen (Voice Activity Detection): The system detects when Alex starts speaking and when Alex stops. VAD gates speech boundaries; a separate noise-suppression stage reduces noise mixed with speech.
- Transcribe (Speech-to-Text): The audio is converted to text. This must happen while Alex is still speaking (streaming), rather than waiting until Alex finishes.
- Think (Large Language Model): The model may prepare from stable partial transcripts, but the orchestrator must not speak a factual answer or execute a side effect until turn-commit policy accepts the input. Once committed, it streams the answer in speakable chunks.
- Speak (Text-to-Speech): The voice synthesis engine reads the first few words as soon as they arrive from the LLM, without waiting for the full sentence.
The first design question is timing, not model brand: which work can overlap before Alex hears a response, and which work must wait for a committed turn? Set that budget before choosing components.
The latency budget: a worked example
The incident line needs a near-full-duplex call: Alex can interrupt at any time, the agent looks up incident state, and first audio still lands inside the measured budget.
What Alex's call has to do
- Keep a near-full-duplex line: Alex can interrupt while the agent is speaking.
- Turn speech into a committed incident query, not a guess from the first partial.
- Look up incident 4219 and summarize severity without blocking the first spoken clause.
- Start speaking as soon as a stable clause exists.
- Mute playback the moment Alex talks over the answer.
Example service objectives
- Speech-end to first audio: Track p50 and p95 from the acoustic end of user speech to first played assistant audio. This is the user-visible end-to-end clock; an operations line might start with a p95 target of 500ms on declared network slices.
- Commit to first audio: Track the internal clock from accepted turn commitment to first played audio. A route might start with a p95 target of 400ms. This excludes endpointing delay and makes post-commit LLM, TTS, and playout regressions easier to isolate.
- Mute on barge-in: Track user-speech onset to muted playback; a product might set a p95 target of 100ms.
- Concurrency: Size media and inference tiers from forecast active sessions and measured per-session work, rather than assuming
10k+. - Audio path: Negotiate supported codecs and sample rates, then measure jitter, loss, and intelligibility by device class.
- Availability: Choose an error budget for the call product and include media-connect, model, and tool failure modes.
Latency budget breakdown
Use 500ms speech-end TTFA and 400ms commit-to-audio as example targets, not laws of physics. A deployed route must report both distributions rather than promise one illustrative trace.
Trace the ranges from one shared event: acoustic speech end at 0ms. Streaming STT has already been consuming frames during the utterance, so only its finalization tail remains after speech ends. Read-only preparation may also be warm before commit. TTS starts only after an accepted turn yields a stable speakable clause.
Keep the two clocks separate. Speech-end TTFA includes endpointing and commit delay. Commit-to-audio starts at the accepted boundary and measures the internal response path. Streaming STT begins during speech rather than waiting for either clock.
Alex's "What is incident status?" query shows the timing path with real numbers:
| Stage | Position relative to acoustic speech end | What's happening |
|---|---|---|
| Streaming STT partials | Begin during speech, often before 0ms | Decoder emits "What is incident" while Alex is still talking |
| Turn commit | 0-60ms | Semantic end-of-turn (or hybrid policy) accepts that Alex's turn is complete; not a pure silence-threshold wait |
| STT finalization tail | 0-100ms | Final transcript catches up with audio already streamed during speech |
| First speakable LLM tokens / clause fragment | 60-240ms | Post-commit generation streams tokens; a speakable fragment can be ready before the full clause ends |
| First TTS chunk | 210-345ms | Synthesizer starts on the first speakable fragment (here at 210ms), while the clause may still finish at 240ms |
| Playout buffer | 345-380ms | Jitter buffer smooths network variance before the speaker |
If the post-speech spans ran one after another, their durations would add to about 510ms. In practice:
- STT started decoding before Alex finished the word "incident," so its full decoding cost isn't added after speech end.
- The orchestrator can pre-classify intent or start a cancellable read-only lookup from stable words; it can't commit an incident number, speak a result, or perform a side effect from an unstable transcript.
- After turn commitment, TTS starts synthesizing as soon as the LLM streams a speakable fragment (for example the start of "I found that incident"), before the full clause is finished. This overlap lets the TTS window open at 210ms while the LLM clause window still ends at 240ms.
- The
0-60mscommit window assumes semantic end-of-turn or an early-commit policy, not a fixed ~300ms silence threshold alone. Pure silence endpointing that waits ~300ms after speech end can't hit a 60ms commit; use the longer silence budget only when semantic EOT is unavailable or unreliable.
This candidate trace plays first audio 380ms after acoustic speech end and 320ms after turn commit. Those numbers describe one path, not a rollout decision. Ship only after measuring p50/p95 for both clocks alongside incorrect-commit rate, endpointing aggressiveness, model size, and network jitter.
1import json
2
3# One measured trace, in milliseconds after acoustic speech end.
4spans = {
5 "endpoint_commit": (0, 60),
6 "stt_finalization": (0, 100),
7 "llm_first_clause": (60, 240),
8 "tts_first_chunk": (210, 345),
9 "playout_buffer": (345, 380),
10}
11speech_end_ms = 0
12turn_commit_ms = spans["endpoint_commit"][1]
13first_audio_ms = spans["playout_buffer"][1]
14speech_end_target_ms = 500
15commit_target_ms = 400
16
17print(json.dumps({
18 "sequential_sum_ms": sum(end - start for start, end in spans.values()),
19 "speech_end_to_first_audio_ms": first_audio_ms - speech_end_ms,
20 "commit_to_first_audio_ms": first_audio_ms - turn_commit_ms,
21 "speech_end_target_ms": speech_end_target_ms,
22 "commit_target_ms": commit_target_ms,
23 "meets_speech_end_trace_target": first_audio_ms - speech_end_ms <= speech_end_target_ms,
24 "meets_commit_trace_target": first_audio_ms - turn_commit_ms <= commit_target_ms,
25 "ship_decision_requires_p95": True,
26}, indent=2))1{
2 "sequential_sum_ms": 510,
3 "speech_end_to_first_audio_ms": 380,
4 "commit_to_first_audio_ms": 320,
5 "speech_end_target_ms": 500,
6 "commit_target_ms": 400,
7 "meets_speech_end_trace_target": true,
8 "meets_commit_trace_target": true,
9 "ship_decision_requires_p95": true
10}Why is the sequential latency sum misleading?
Answer
Production voice pipelines overlap stages. STT starts during speech, reversible preparation can start from stable partials, and TTS can synthesize the first accepted clause before the full answer is done. Measure speech-end-to-audio for user experience and commit-to-audio for the internal response path.
How the pieces fit together
The architecture usually separates three concerns: client-side capture and playback, a latency-sensitive media edge, and the inference pipeline. Picture a call where the LLM p95 stays flat but mobile callers begin hearing gaps. That points first to packet timing, echo, or playout state at the media edge, not to model execution. Decoupling transport from inference makes that diagnosis possible.
In many deployments, a media gateway terminates WebRTC (Web Real-Time Communication) or SIP (Session Initiation Protocol), handles jitter buffers and routing, then forwards audio to inference over an internal low-latency link.[4] Not every system needs a full SFU (Selective Forwarding Unit) for a 1:1 assistant, but most serious deployments still benefit from a dedicated ingress tier close to the user.
Within inference, the voice pipeline orchestrates turn detection, STT or native-audio encoding, LLM reasoning, tool calls, and TTS or audio decoding. A conversation state manager sits alongside the pipeline, keeping track of dialogue turns, client playout offsets, and tool results so the model only commits to what the user heard.
The top row is the hot path. The bottom lanes are control planes: mute as soon as local VAD hears Alex, then record only audio whose playout the client acknowledged. A healthy model can't repair a media path that has already missed its playout deadline.
Why split media transport from inference orchestration?
Answer
Audio transport handles jitter, packet loss, echo, routing, and playback timing. Inference handles STT, tools, LLM, and TTS. Their failure modes and scaling profiles differ.
Knowing when to listen: voice activity detection
Voice Activity Detection (VAD) gives the system frame-level evidence that the user is speaking. In Alex's incident-status call, that evidence can start capture and trigger a fast mute, but it can't by itself say that Alex's question is complete.
VAD isn't the same thing as turn detection. VAD classifies each audio frame as speech or silence. Turn detection (endpointing) decides when the user's turn has finished so the agent can answer. Modern stacks increasingly layer three signals: VAD for raw speech presence, transcript-level endpointing for silence after stable words, and a model that judges semantic completeness from the partial transcript. Model-based end-of-turn detectors can commit a turn before trailing silence accumulates, cutting both clipped speakers and laggy responses.[5] You still keep a fast VAD underneath for instant barge-in.
Test the boundary with Alex saying, "incident four ... two ... one ... nine" and pausing between digits. Frame-level VAD may see silence after four; an endpoint that commits there sends an incomplete identifier downstream. Semantic endpointing can hold the turn through those pauses, then commit once the request is complete. The faster signal should mute playback, while the richer signal owns transcript and tool commitment.
Client-side VAD is common because it gives instant barge-in detection and avoids sending obvious silence. Server-side turn detection is also common because it centralizes tuning and can use richer context. This local wrapper represents a streaming VAD backend such as Silero VAD[6].
1from collections import deque
2from dataclasses import asdict, dataclass
3import json
4
5@dataclass
6class VADEvent:
7 type: str
8 frame: int
9 audio_ms: int
10
11class VoiceActivityDetector:
12 """Detect speech boundaries from one probability per 20ms audio frame."""
13
14 def __init__(
15 self,
16 frame_ms: int = 20,
17 speech_threshold: float = 0.5,
18 silence_duration_ms: int = 60,
19 prefix_padding_ms: int = 40,
20 ):
21 self.frame_ms = frame_ms
22 self.speech_threshold = speech_threshold
23 self.silence_frames_required = max(1, silence_duration_ms // frame_ms)
24 self.pre_roll = deque(maxlen=max(1, prefix_padding_ms // frame_ms))
25 self.speech_frames: list[int] = []
26 self.in_speech = False
27 self.trailing_silence = 0
28
29 def process_frame(self, frame_id: int, probability: float) -> VADEvent | None:
30 """Return speech boundary events for one audio frame."""
31 if probability >= self.speech_threshold:
32 if not self.in_speech:
33 self.in_speech = True
34 self.speech_frames = list(self.pre_roll)
35 self.speech_frames.append(frame_id)
36 self.trailing_silence = 0
37 return VADEvent("speech_start", frame_id, len(self.speech_frames) * self.frame_ms)
38 self.speech_frames.append(frame_id)
39 self.trailing_silence = 0
40 return None
41
42 if not self.in_speech:
43 self.pre_roll.append(frame_id)
44 return None
45
46 self.speech_frames.append(frame_id)
47 self.trailing_silence += 1
48 if self.trailing_silence < self.silence_frames_required:
49 return None
50
51 utterance_frames = self.speech_frames[:-self.trailing_silence]
52 self.in_speech = False
53 self.speech_frames = []
54 self.trailing_silence = 0
55 self.pre_roll.clear()
56 return VADEvent("speech_end", frame_id, len(utterance_frames) * self.frame_ms)
57
58probabilities = [0.04, 0.08, 0.66, 0.74, 0.69, 0.22, 0.15, 0.09]
59detector = VoiceActivityDetector()
60events = []
61
62for frame_id, probability in enumerate(probabilities):
63 event = detector.process_frame(frame_id, probability)
64 if event:
65 events.append(asdict(event))
66
67print(json.dumps(events, indent=2))1[
2 {
3 "type": "speech_start",
4 "frame": 2,
5 "audio_ms": 60
6 },
7 {
8 "type": "speech_end",
9 "frame": 7,
10 "audio_ms": 100
11 }
12]The demo uses a short silence_duration_ms = 60 so its output fits in eight frames. A phone support bot might begin testing around 300ms, then tune from measured false cutoffs and lag. Slow speakers may need longer thresholds; push-to-talk or command-style assistants can go shorter. A fixed silence threshold is the floor, not the ceiling: a semantic end-of-turn model can hold through mid-sentence pauses (a caller reciting a phone number digit by digit) and fire early once the utterance is complete, so you don't have to pick one silence value that hurts every other case.[5]
VAD deployment: client vs. server
Choosing where to run VAD changes responsiveness, bandwidth, and operational control. Client-side detection wins on local responsiveness. Server-side detection wins on centralized tuning and consistent turn boundaries across devices.
| Feature | Client-Side VAD | Server-Side Detection |
|---|---|---|
| Barge-in reaction | Can mute local playout without a network round trip | Includes transport delay before server action reaches playback |
| Bandwidth | Can suppress silence if upload policy uses its output | Receives the configured upstream stream continuously |
| Operational control | Requires testing across device/browser classes | Centralizes threshold and model tuning |
| Turn consistency | Device signals may differ | Can apply one server-side commit policy |
| Data exposure | Reduced only if capture/upload policy withholds silent frames | Depends on retained audio and server policy |
Many production systems use both: lightweight local VAD to mute playback instantly on barge-in, plus server-side turn detection to decide when transcripts are stable enough to trigger response generation. Measure disagreements by device class. A local false positive is an interruption defect; a server false positive can truncate a number or fire a tool.
The hybrid setup gives the user the fast interruption path they expect while keeping final turn boundaries consistent enough for transcripts, tools, and analytics. It also gives you two timestamps to compare when a call feels wrong: speech onset for mute latency, and committed endpoint for response latency.
Why run both local VAD and server-side turn detection?
Answer
Local VAD can mute local playback quickly and can suppress silence before upload when configured to do so. Server-side detection (silence endpointing plus a semantic end-of-turn model) gives centralized boundaries and richer context for deciding when the turn is truly complete, beyond the moment audio goes quiet.
Acoustic echo cancellation (AEC)
Acoustic echo cancellation (AEC) prevents the agent from hearing itself. When the agent speaks and the user interrupts, the microphone picks up both the user's voice and the agent's own output playing from the speaker. Without AEC, the VAD would trigger on the agent's voice, creating an echo loop where the agent transcribes its own speech and responds to it.
Modern browser capture stacks expose echo-cancellation constraints so applications can request microphone input with system or remote audio removed before it reaches VAD and STT.[7] The browser or device owns that processing, and implementations vary. Treat the constraint as a request, then test the resulting audio path across speaker volume, microphone placement, rooms, and device classes.
One common failure is deploying voice agents without AEC testing in real acoustic environments. Laptop speakers and microphones placed close together, as in mobile phones, create strong echo that untested AEC configurations may fail to suppress. The symptom is brutal: the agent interrupts itself.
What happens if echo cancellation fails during agent playback?
Answer
The microphone can capture the agent's own voice, VAD treats it as user speech, STT transcribes it, and the agent can interrupt or respond to itself.
From sound to text: speech-to-text
Two primary approaches exist for real-time STT. In Alex's call, streaming STT lets the system recognize "What is incident" before Alex finishes saying "incident." That early text is useful only if the orchestrator knows which actions are safe to reverse.
Streaming STT (recommended for latency)
Streaming STT sends partial transcripts as audio arrives, allowing the LLM to pre-read the turn and start cancellable, read-only preparation before the utterance is complete. Provider SDKs differ, but the state machine must separate unstable interim text from committed text that may enter history, trigger tools, or reach TTS.
1from dataclasses import asdict, dataclass
2import json
3
4@dataclass
5class TranscriptEvent:
6 kind: str
7 text: str
8 stable_prefix: str
9
10def common_prefix(left: str, right: str) -> str:
11 shared = []
12 for left_word, right_word in zip(left.split(), right.split()):
13 if left_word != right_word:
14 break
15 shared.append(left_word)
16 return " ".join(shared)
17
18def emit_transcript_events(partials: list[tuple[str, bool]]) -> list[TranscriptEvent]:
19 events = []
20 previous_text = ""
21
22 for text, is_final in partials:
23 stable_prefix = text if is_final else common_prefix(previous_text, text)
24
25 if is_final:
26 events.append(TranscriptEvent("final", text, stable_prefix))
27 else:
28 events.append(TranscriptEvent("interim", text, stable_prefix))
29 previous_text = text
30
31 return events
32
33partials = [
34 ("What is", False),
35 ("What is incident", False),
36 ("What is incident number four two", False),
37 ("What is incident number four two one nine?", True),
38]
39
40print(json.dumps([asdict(event) for event in emit_transcript_events(partials)], indent=2))1[
2 {
3 "kind": "interim",
4 "text": "What is",
5 "stable_prefix": ""
6 },
7 {
8 "kind": "interim",
9 "text": "What is incident",
10 "stable_prefix": "What is"
11 },
12 {
13 "kind": "interim",
14 "text": "What is incident number four two",
15 "stable_prefix": "What is incident"
16 },
17 {
18 "kind": "final",
19 "text": "What is incident number four two one nine?",
20 "stable_prefix": "What is incident number four two one nine?"
21 }
22]The shared-prefix heuristic is intentionally conservative: it only exposes words repeated across two successive hypotheses. Even that prefix is tentative because a provider can revise earlier words later. Ask what should happen when the partial reads Close incident four two one nine: classify it and prefetch a read-only record, but don't write history, speak a result, or close anything. The commit boundary matters most when a partial transcript changes meaning. An incident lookup may be safe to prefetch and discard; closing an incident or promising mitigation status isn't.
1import json
2
3def permitted_actions(text: str, committed: bool) -> list[str]:
4 actions = ["classify_intent"]
5 if "incident" in text.lower():
6 actions.append("prefetch_read_only_status")
7 if committed:
8 actions.extend(["write_history", "speak_response"])
9 if "close" in text.lower():
10 actions.append("ask_for_close_confirmation")
11 return actions
12
13events = [
14 {"text": "Close incident four two one nine", "committed": False},
15 {"text": "Show incident four two one nine", "committed": True},
16]
17
18print(json.dumps([
19 {"committed": event["committed"], "actions": permitted_actions(**event)}
20 for event in events
21], indent=2))1[
2 {
3 "committed": false,
4 "actions": [
5 "classify_intent",
6 "prefetch_read_only_status"
7 ]
8 },
9 {
10 "committed": true,
11 "actions": [
12 "classify_intent",
13 "prefetch_read_only_status",
14 "write_history",
15 "speak_response"
16 ]
17 }
18]End-of-utterance STT (simpler, slightly higher latency)
Using a batch recognizer such as Whisper[8] after an endpoint event is much simpler, but it adds the endpoint wait and full decode after the user stops talking.
1import json
2
3def batch_stt_timeline(speech_ms: int, endpoint_ms: int, decode_ms: int) -> dict[str, int | str]:
4 acoustic_speech_end_ms = speech_ms
5 turn_commit_ms = acoustic_speech_end_ms + endpoint_ms
6 transcript_ready_ms = turn_commit_ms + decode_ms
7 return {
8 "approach": "end-of-utterance",
9 "speech_ms": speech_ms,
10 "endpoint_ms": endpoint_ms,
11 "decode_ms": decode_ms,
12 "acoustic_speech_end_ms": acoustic_speech_end_ms,
13 "turn_commit_ms": turn_commit_ms,
14 "first_transcript_ms": transcript_ready_ms,
15 }
16
17print(json.dumps(batch_stt_timeline(speech_ms=1200, endpoint_ms=300, decode_ms=450), indent=2))1{
2 "approach": "end-of-utterance",
3 "speech_ms": 1200,
4 "endpoint_ms": 300,
5 "decode_ms": 450,
6 "acoustic_speech_end_ms": 1200,
7 "turn_commit_ms": 1500,
8 "first_transcript_ms": 1950
9}| Approach | Latency | Complexity |
|---|---|---|
| Streaming STT | Illustrative first partial: ~50-150ms | High |
| End-of-utterance | Illustrative post-end decode: ~200-800ms+ | Low |
When is batch STT acceptable, and when is it a bad fit?
Answer
Batch STT is acceptable for non-live recordings or low-complexity prototypes. It's a bad fit for natural conversation because it waits until the user stops before decoding.
Speak a first clause, not a full plan
After turn commitment, the LLM should emit a stable first speakable clause without waiting for a long response plan. Before commitment, the system may only do reversible preparation. That makes Time-to-First-Token (TTFT) relevant without letting an unstable transcript become spoken truth. The serving mechanics are in Inference Mechanics.
Long reasoning and slow tools are a poor fit for the acknowledgement path unless measurements still meet the TTFA objective. The agent can say an honest status quickly, run the slow work in the background, and speak the result when it lands.
Give the first clause one job: tell Alex something true and useful now. Use short clauses, no Markdown, tool status early, and chunks TTS can consume before the full response is complete. A long plan can follow after the call has regained its rhythm.
1import json
2
3def voice_chunks(transcript: str, incident_status: dict[str, str]) -> list[str]:
4 reply = (
5 f"Incident {incident_status['id']} is still open. "
6 f"Severity is {incident_status['severity']}. "
7 "I can text you the incident link."
8 )
9 clauses = [part.strip() + "." for part in reply.split(".") if part.strip()]
10 return clauses
11
12transcript = "What is incident number four two one nine?"
13chunks = voice_chunks(
14 transcript=transcript,
15 incident_status={"id": "4219", "severity": "sev-two"},
16)
17
18print(json.dumps({
19 "contains_incident": "incident" in transcript.lower(),
20 "max_words_per_clause": max(len(clause.split()) for clause in chunks),
21 "streamed_clauses": chunks,
22}, indent=2))1{
2 "contains_incident": true,
3 "max_words_per_clause": 7,
4 "streamed_clauses": [
5 "Incident 4219 is still open.",
6 "Severity is sev-two.",
7 "I can text you the incident link."
8 ]
9}Prompt engineering for voice is distinct from text. Voice prompts must forbid Markdown, which TTS reads poorly, and front-load the answer so the first clause is worth hearing.
Slow tools don't get to steal the first word
Incident lookup can often finish inside the commit-to-audio budget, especially if you prefetched a read-only status from a stable partial. Paging an owner, opening a ticket, or waiting on a human ack can't. If the side effect won't return before first audio is due, speak an honest acknowledgement and finish the tool off the hot path. Don't narrate a result you don't have yet.
Give the helper a committed intent and a measured tool latency. Fast reads may speak the fact. Slow writes speak a status and keep the tool running. Confirmation stays a separate boundary: a fast tool still mustn't turn a request to close into an unapproved side effect.
1import json
2
3COMMIT_TO_AUDIO_BUDGET_MS = 400
4
5def first_speakable(intent: str, tool_ms: int, incident_id: str) -> dict[str, str | bool]:
6 if intent == "lookup_status" and tool_ms <= COMMIT_TO_AUDIO_BUDGET_MS:
7 return {
8 "speak": f"Incident {incident_id} is still open.",
9 "tool_blocking": False,
10 "side_effect": "none",
11 }
12 if intent == "page_owner":
13 if tool_ms <= COMMIT_TO_AUDIO_BUDGET_MS:
14 return {
15 "speak": "I paged the owner.",
16 "tool_blocking": False,
17 "side_effect": "page_owner",
18 }
19 return {
20 "speak": "Paging the owner now.",
21 "tool_blocking": False,
22 "side_effect": "page_owner_async",
23 }
24 if intent == "close_incident":
25 return {
26 "speak": "I can close that incident. Should I?",
27 "tool_blocking": False,
28 "side_effect": "none_until_confirm",
29 }
30 return {
31 "speak": "I heard you. Give me a moment.",
32 "tool_blocking": False,
33 "side_effect": "none",
34 }
35
36print(json.dumps([
37 first_speakable("lookup_status", 180, "4219"),
38 first_speakable("page_owner", 2400, "4219"),
39 first_speakable("close_incident", 90, "4219"),
40], indent=2))1[
2 {
3 "speak": "Incident 4219 is still open.",
4 "tool_blocking": false,
5 "side_effect": "none"
6 },
7 {
8 "speak": "Paging the owner now.",
9 "tool_blocking": false,
10 "side_effect": "page_owner_async"
11 },
12 {
13 "speak": "I can close that incident. Should I?",
14 "tool_blocking": false,
15 "side_effect": "none_until_confirm"
16 }
17]Why should voice prompts forbid Markdown and long setup?
Answer
TTS reads markdown awkwardly, and long preambles delay useful audio. Voice responses should front-load the answer in short speakable clauses.
What should the agent say if paging the owner takes 2.4 seconds?
Answer
Speak an honest acknowledgement such as "Paging the owner now," run the page asynchronously, and don't claim the page succeeded until the tool result arrives.
Serving optimizations that matter in voice
Live voice sessions stress inference differently from chat. TTFT matters more than long-form throughput because TTS can't speak until the first stable clause arrives. Session length also matters because a 20-minute call can accumulate enough context to make KV-cache management part of the latency budget, not a background infrastructure concern. If TTFT rises only on long calls, inspect context growth and cache pressure before swapping models.
- Paged KV caches: Long-lived sessions fragment memory if every request expects one contiguous KV-cache region. PagedAttention-style allocators let serving systems grow and trim per-session context without constant copying, which helps keep latency stable across many simultaneous calls.[9]
- Speculative decoding: A draft model proposes tokens that the target model verifies in parallel, reducing decode latency when acceptance and implementation overhead are favorable.[10] It isn't an automatic TTFT improvement; benchmark TTFA and total clause latency on the served model pair.
- Aggressive context hygiene: Don't keep every interim transcript forever. Preserve final transcripts, tool results, and only the portion of assistant audio the user heard before barge-in.
Prompt caching is prefix economics
Prompt caching only helps when repeated requests preserve a matching prefix. Official OpenAI guidance recommends placing static instructions and examples at the beginning and variable content later.[11] Voice sessions are good candidates because system policy, voice style, and tool schemas often stay fixed across many turns.
Build the prompt in this order:
- Stable policy, output style, and safety instructions.
- Stable tool definitions and few-shot examples.
- Conversation history that grows by appending committed turns.
- Dynamic call state such as current incident result, timestamp, or playback acknowledgement.
Putting current_time=... or a rolling account balance near the top changes early tokens and can turn every later request into a cache miss. Keep volatile facts in a suffix or tool result, but never move instructions merely to chase hits when doing so changes model behavior or the authorization boundary.
Measure economics directly:
Provider eligibility, retention, and cached-input rates change, so read the usage metadata and current official pricing rather than assuming one discount. Track eligible prefix tokens, cached tokens, the cache-hit ratio, uncached input cost per completed call, and TTFA. A cache hit can lower repeated prefill work; it doesn't fix endpointing, tool, TTS, or playout latency.
Why should the current incident status appear after stable voice instructions and tool schemas?
Answer
The status changes often, so placing it near the top busts the reusable prefix. Keep the stable policy first and the dynamic state later, then verify the usage metadata, cost, latency, and unchanged behavior.
Why should interim transcripts not live forever in voice-agent context?
Answer
Interim text can be wrong and bloats long sessions. Keep final transcripts, tool results, and assistant audio actually heard; drop unstable partials and truncated speech.
Speaking without waiting for the full answer: TTS with streaming
Begin speaking before the full response is generated. The TTS engine receives a stream of text tokens, buffers them into sentences or phrases, and synthesizes audio chunks on the fly. The first chunk should answer the immediate question; later chunks can carry detail.
Waiting for the LLM to finish an entire sentence or paragraph before starting TTS can add seconds of latency. Stream tokens and synthesize clause-by-clause instead.
There are two bad schedules to predict: a full paragraph delays first audio, while one TTS call per token makes the voice stutter. Buffer punctuation or a short character limit, then measure whether the clause sounds natural and arrives inside TTFA.
1import json
2
3def flush_speakable_clauses(tokens: list[str], max_chars: int = 52) -> list[str]:
4 buffer = ""
5 flushed = []
6
7 for token in tokens:
8 buffer += token
9 stripped = buffer.strip()
10 if stripped.endswith((",", ".", "!", "?")) or len(stripped) >= max_chars:
11 flushed.append(stripped)
12 buffer = ""
13
14 if buffer.strip():
15 flushed.append(buffer.strip())
16
17 return flushed
18
19tokens = [
20 "Incident ", "4219 ", "is ", "still ", "open, ",
21 "severity ", "is ", "sev-two. ",
22 "I ", "can ", "text ", "the ", "incident ", "link."
23]
24
25print(json.dumps(flush_speakable_clauses(tokens), indent=2))1[
2 "Incident 4219 is still open,",
3 "severity is sev-two.",
4 "I can text the incident link."
5]How parallel execution reduces latency
The end-to-end target starts at acoustic speech end. The internal target starts at turn commit. Those clocks only close if stages overlap:
- VAD runs continuously, watching frames as they arrive and contributing a speech-end signal to endpointing.
- Streaming STT is already running during speech. Silence doesn't start recognition; it helps the endpoint policy decide when the current turn may commit.
- Reversible LLM preparation can start from stable STT partials. Spoken or side-effecting work still waits for turn commitment.
- TTS begins synthesizing as soon as the committed response has a stable speakable clause, which may be well before the LLM finishes.
- Audio playback begins as soon as the first TTS chunk is ready.
On the incident trace, first audio is 380ms after speech end. If the same span durations ran one after another, first audio would be 510ms. The gain comes from schedule overlap: STT works during speech, and TTS starts while the LLM is still finishing its clause.
Why synthesize clauses instead of individual tokens or full paragraphs?
Answer
Token-level TTS sounds choppy, while paragraph-level TTS adds seconds. Clause buffering gives natural phrasing while keeping first audio fast.
When the user interrupts: barge-in handling
When the user speaks over the agent ("barge-in"), the system has to react immediately. Alex asks about an incident, the agent starts explaining mitigation, and Alex cuts in with "Wait, page the owner." The agent must stop talking, cancel leftover generation, and remember only what actually played. Treat this as a state transition, not as a UI mute: the next turn must start from the heard boundary.
Local VAD fires the mute. Server endpointing still decides when the new user turn is committed. Those signals form one control loop: fast local action first, authoritative transcript boundary second.
- Stop Playback: Halt the audio output. A support product might begin with a p95 mute target below 100ms, then tune against measured paths.
- Cancel Generation: Abort in-flight TTS and LLM requests to save cost and compute.
- State Reconciliation: Update conversation context to reflect only what the user heard.
Stopping the speaker is only half of barge-in. Suppose Alex interrupts at 650ms and the latest client playout acknowledgement reports a cursor of 520ms. The next model call should remember the clause through that cursor, not the mitigation text that was still buffered or in flight.
Use conservative reconciliation at audio-chunk boundaries. When VAD detects fresh user speech during agent playback, stop playout, cancel in-flight generation, and truncate assistant text to the portion confirmed by the latest client playout acknowledgement. Finer timestamps can reduce the amount discarded, but they must still describe playout, not output receipt or generation.
1from dataclasses import dataclass
2import json
3
4@dataclass
5class PlayoutChunk:
6 text: str
7 text_end_char: int # Exclusive end offset in pending_text.
8 playout_end_ms: int
9
10class InterruptionHandler:
11 def __init__(self, pending_text: str, playout_chunks: list[PlayoutChunk]):
12 self.pending_text = pending_text
13 self.playout_chunks = playout_chunks
14
15 def reconcile(self, playout_cursor_ms: int) -> dict[str, object]:
16 heard_chars = self._chars_heard(playout_cursor_ms)
17 spoken_text = self.pending_text[:heard_chars].rstrip()
18 discarded_text = self.pending_text[heard_chars:].strip()
19 return {
20 "history_to_keep": [{"role": "assistant", "content": spoken_text}],
21 "discarded_generated_text": discarded_text,
22 }
23
24 def _chars_heard(self, playout_cursor_ms: int) -> int:
25 heard_chars = 0
26 for chunk in self.playout_chunks:
27 if chunk.playout_end_ms <= playout_cursor_ms:
28 heard_chars = chunk.text_end_char
29 else:
30 break
31 return heard_chars
32
33pending = "Incident is open because the database failover is still running."
34chunks = [
35 PlayoutChunk("Incident is", 11, 240),
36 PlayoutChunk(" open", 16, 520),
37 PlayoutChunk(" because the database", 37, 860),
38 PlayoutChunk(" failover is still running.", 64, 1120),
39]
40
41# At the 650ms interrupt, the latest client acknowledgement is 520ms.
42result = InterruptionHandler(pending, chunks).reconcile(playout_cursor_ms=520)
43print(json.dumps(result, indent=2))1{
2 "history_to_keep": [
3 {
4 "role": "assistant",
5 "content": "Incident is open"
6 }
7 ],
8 "discarded_generated_text": "because the database failover is still running."
9}The barge-in handler keeps only what the client confirmed through its playout cursor. text_end_char is an exclusive Python slice boundary: offset 16 keeps pending_text[:16], which is "Incident is open". A live media path should use smaller chunks, or map synthesized spans to client acknowledgements, playout timestamps, or RTP sequence ranges, when it needs finer reconciliation. Don't rely on model tokens or output receipt alone. Jitter buffers can delay or drop chunks after synthesis, so "generated" isn't the same as confirmed playout.
Why must barge-in history use "heard" text, not generated text?
Answer
The user may interrupt before playback finishes. If history includes unheard generated text, the model assumes shared context that the user never received, causing confusing follow-ups.
Cascaded pipeline vs. native audio
Design reviews often ask you to compare two families: cascaded STT -> LLM -> TTS pipelines and native speech-to-speech models. Early research systems such as AudioPaLM and SpeechGPT showed the idea was viable.[2][3] Native audio is now a production API option. OpenAI's Realtime API supports live speech-to-speech sessions.[12] Google's Gemini Live API is a stateful WebSocket session with native audio, automatic voice-activity detection, barge-in, transcripts, and tool use.[13]
Choose transport from where the audio lives: WebRTC for browser and mobile capture, a WebSocket session when your server already has PCM, and SIP when the caller is on the phone network. If the speech model only speaks WebSocket, terminate WebRTC at the media edge and forward PCM. Provider model catalogs change quickly, so verify current aliases when you implement the route.
Native models reduce model handoffs and can retain prosodic signals that a text-only boundary loses. They don't remove transport, VAD, AEC, or state-management problems. Gemini documents barge-in plus input and output transcripts.[13] On interrupt, keep only audio that reached the client and was confirmed by its playout cursor, then discard the rest. That's the same heard-versus-generated rule the cascaded path has to implement in your orchestrator.
Make the choice with a failure question: if a caller disputes what the agent said, can you inspect the transcript, tool decision, and exact played audio? Cascading exposes those boundaries and makes component swaps easier. Native audio may shorten the model path, but it still needs those side channels when policy, auditability, or deterministic tools matter.
Use the comparison as a control checklist. Native audio can shorten the model path, but you still need stateful sessions, transcript side channels, tool policy, and interruption handling. Measure that whole path before calling a handoff reduction a latency win.
⚠️ Common mistake: Treating a WebSocket speech session as a replacement for WebRTC. Gemini Live, and any other server WebSocket model, still needs a media edge that handles jitter, echo, and device capture. The speech model is not the transport.
Many commercial native-audio APIs use stateful sessions, not one-shot request/response calls. The client streams audio frames in, then receives incremental audio and transcript events back.
Session shape
In this simplified session loop, an audio_delta means output audio reached the client buffer; it doesn't prove that the speaker played it. Track received output separately, and advance a confirmed_playout_ms cursor only from an application-level playback_ack emitted after the client audio sink consumes that range. OpenAI's Realtime client event conversation.item.truncate uses audio_end_ms to synchronize server history with this client-side playback boundary.[14]
1from collections import defaultdict
2import json
3
4events = [
5 {"type": "transcript_delta", "text": "Incident is"},
6 {"type": "audio_delta", "audio_ms": 160},
7 {"type": "playback_ack", "playout_cursor_ms": 160},
8 {"type": "transcript_delta", "text": " still open."},
9 {"type": "audio_delta", "audio_ms": 220},
10 {"type": "tool_call", "name": "send_incident_link"},
11]
12
13state = defaultdict(list)
14received_audio_ms = 0
15confirmed_playout_ms = 0
16
17for event in events:
18 if event["type"] == "audio_delta":
19 received_audio_ms += event["audio_ms"]
20 elif event["type"] == "playback_ack":
21 cursor = event["playout_cursor_ms"]
22 if cursor < confirmed_playout_ms or cursor > received_audio_ms:
23 raise ValueError("playback acknowledgement must be monotonic and <= received audio")
24 confirmed_playout_ms = cursor
25 elif event["type"] == "transcript_delta":
26 state["transcript"].append(event["text"])
27 elif event["type"] == "tool_call":
28 state["tool_calls"].append(event["name"])
29
30print(json.dumps({
31 "transcript_side_channel": "".join(state["transcript"]),
32 "received_audio_ms": received_audio_ms,
33 "confirmed_playout_ms": confirmed_playout_ms,
34 "received_not_confirmed_ms": received_audio_ms - confirmed_playout_ms,
35 "tool_calls": state["tool_calls"],
36}, indent=2))1{
2 "transcript_side_channel": "Incident is still open.",
3 "received_audio_ms": 380,
4 "confirmed_playout_ms": 160,
5 "received_not_confirmed_ms": 220,
6 "tool_calls": [
7 "send_incident_link"
8 ]
9}The client received 380ms of output, but its acknowledgement confirms only 160ms consumed by the speaker. The remaining 220ms may still be buffered or cancelled, so it isn't eligible for conversation history. Barge-in reconciliation uses the same confirmed-playout boundary.
Audio tokenization
Published native-audio systems commonly avoid feeding raw PCM directly into a transformer. AudioPaLM and SpeechGPT use compressed speech representations rather than modeling every waveform sample as its own sequence element.[2][3] The practical question is sequence length: a representation that preserves useful timing and tone while reducing steps gives the model a tractable stream.
Generation works in reverse: the model emits audio tokens or latents, a decoder reconstructs waveform chunks, and the client plays them with a jitter buffer. System-design point: compressed speech representations make sequence modeling practical while preserving tone, pacing, and other paralinguistic cues.
Trade-offs
Native audio can avoid some model handoffs and retain paralinguistic features such as tone, hesitation, and emphasis that a strict text bottleneck removes. Whether it produces faster or better interactions is an evaluation result, not an architectural guarantee.
Native audio is also harder to debug and harder to control. Without a clean intermediate transcript, it's tougher to inspect failures, run deterministic tool policies, or align partially spoken output with conversation history. Modular pipelines are still easier to observe, test, and swap component by component.
When should you prefer a cascaded STT-LLM-TTS pipeline over native audio?
Answer
Prefer cascaded pipelines when observability, deterministic tool policy, component swaps, transcript inspection, or compliance logs matter more than maximum prosody and theoretical latency.
Moving audio across the internet
When moving live audio across the internet, transport choice sets baseline latency and jitter. For browser and mobile voice agents, WebRTC prefers a media-timed UDP path when available, but ICE/TURN deployments need relay and TCP/TLS fallback paths for restricted networks (RFC 8835).
WebSockets (persistent, full-duplex TCP connections) remain useful for signaling, transcription-only streams, or server-to-server audio when you're willing to manage media buffering yourself. They don't become a media transport merely because audio bytes fit inside the connection.
TCP provides reliable, ordered byte delivery with sequence numbers, acknowledgments, and retransmission. That's great for text and file transfer, but it creates head-of-line blocking for live audio. If one segment is lost, later bytes wait behind recovery. In voice, that often shows up as stuttery playout or latency spikes.
On a WebRTC UDP media path, loss handling is deadline-aware: the stream can keep moving while packet-loss concealment fills short gaps. For conversation, concealing a short lost frame may be preferable to delaying later audio for recovery; verify this trade-off under the network slices you serve.
The deadline changes the protocol choice. RTP recovery helps only while a packet can still improve playout; TCP recovery must preserve byte order even if the voice moment has already passed. If a call is clean on wired networks but stutters on a mobile slice, compare packet arrival, buffer depth, concealment, and playout delay before blaming the model.
Jitter buffers and packet loss concealment
Network jitter (variance in packet arrival time) is the enemy of smooth audio. A jitter buffer on the receiver side (media server or client) holds incoming packets for a short duration before playback. A team might begin experiments around 20-60ms, but the right value depends on network quality and how much delay your UX can tolerate. Raise it and underruns may fall while conversational latency grows; shrink it and first audio may arrive sooner while gaps become more likely.
- Static Jitter Buffer: Fixed size (e.g., 50ms). Simple but risks underruns if jitter exceeds buffer size.
- Adaptive Jitter Buffer: Dynamically resizes based on network conditions. Expands during congestion (adding latency) and shrinks during stability (reducing latency).
Browser WebRTC stacks already include adaptive jitter buffers and packet loss concealment (PLC). NetEQ is a well-known WebRTC receiver implementation of this idea.[15] When packets arrive late or disappear, the receiver can stretch buffered audio, extrapolate from recent audio, or ask the decoder to synthesize concealment frames instead of stalling playout.
Protocol comparison: WebRTC vs. WebSocket
| Feature | WebRTC Media (UDP Preferred, Fallbacks Possible) | WebSocket (TCP) |
|---|---|---|
| Transport | UDP preferred; relay or TCP/TLS fallback paths possible | TCP |
| Reliability Model | Media-timed (may use NACK/FEC, but playout stays time-bounded) | Byte-stream reliable (lost bytes block later bytes) |
| Latency Behavior | Low and stable when tuned | Can spike under loss |
| Congestion Control | Media-aware adaptation via RTP/RTCP feedback (implementation-dependent) | Managed by the TCP stack, not by media playout deadlines |
| Encryption | Mandatory (DTLS (Datagram Transport Layer Security) / SRTP (Secure Real-time Transport Protocol)) | TLS (over TCP) |
| Ideal Use Case | Real-time Voice/Video | Chat, Signaling, File Transfer |
WebRTC media is richer than "UDP means drop packets." On UDP media paths it layers RTP (Real-time Transport Protocol) / RTCP (RTP Control Protocol) feedback, jitter buffers, packet loss concealment (PLC), and sometimes NACK (Negative Acknowledgment) or FEC (Forward Error Correction).[4] ICE and TURN can select relay or fallback paths when direct UDP is unavailable (RFC 8835). The design goal is timeliness: recover loss when it can still help, otherwise keep audio moving.
Using WebSockets as a drop-in replacement for WebRTC in browser live audio creates hidden work. You can ship with WebSockets, but you must own chunking, playout buffering, backpressure, and worse behavior under packet loss.
Why does WebRTC usually beat WebSockets for browser live audio?
Answer
WebRTC uses media-timed transport with jitter buffers, RTP/RTCP feedback, packet loss concealment, and time-bounded playout. TCP WebSockets can stall later audio behind lost bytes.
Scaling to thousands of conversations
Suppose 10,000 callers connect during an incident drill. Connection count alone misses the work each session creates: media state, turn timing, model requests, and bursty GPU demand. The ingress tier usually needs sticky routing or consistent hashing so packets for one conversation keep landing on the same media worker.
The media tier scales by active sessions and network state. Inference scales by model pressure, which is why separating these tiers usually beats one giant "voice server" process. The routing sketch uses a stable modulo hash only to show affinity with a fixed worker set. A production router should define what happens when workers join or fail, often with connection-aware routing or consistent hashing to limit remapping.
Don't turn 10k+ into a capacity promise. Load-test a declared call mix: session duration, utterance and pause lengths, arrival bursts, output lengths, tool rate, codec, and network loss. Record media CPU, GPU queue time, active streams, reconnects, first-audio p50/p95, mute p95, and error rate. The operating point is where those SLOs hold with headroom, not where a dashboard first reaches a session count.
WebRTC connections carry session state. A specific media worker holds Datagram Transport Layer Security (DTLS) keys, Secure Real-time Transport Protocol (SRTP) state, jitter-buffer state, and often the client playout cursor. If a naive round-robin load balancer routes packets to a different worker mid-session, decryption and timing state break immediately.
Behind media workers, inference components should scale independently because their compute profiles differ. Turn detection and some STT stages often fit on CPU. Largest speech or LLM models usually need GPU. Independent scaling keeps GPU utilization high without overprovisioning lighter tiers. If one media worker dies, preserve the call's heard cursor and reconnect state where possible; otherwise make the recovery explicit, rebuild the session, and tell the caller what happened.
1import hashlib
2import json
3
4workers = ["media-a", "media-b", "media-c"]
5
6def route(session_id: str) -> str:
7 digest = int(hashlib.sha256(session_id.encode()).hexdigest(), 16)
8 return workers[digest % len(workers)]
9
10session_id = "call-alex-4219"
11packet_routes = [route(session_id) for _ in range(4)]
12failed_worker = packet_routes[0]
13
14print(json.dumps({
15 "packet_routes": packet_routes,
16 "single_worker_during_session": len(set(packet_routes)) == 1,
17 "failed_worker": failed_worker,
18 "recovery": "reconnect_and_rebuild_session",
19}, indent=2))1{
2 "packet_routes": [
3 "media-c",
4 "media-c",
5 "media-c",
6 "media-c"
7 ],
8 "single_worker_during_session": true,
9 "failed_worker": "media-c",
10 "recovery": "reconnect_and_rebuild_session"
11}Why do WebRTC sessions need sticky routing?
Answer
A media worker holds DTLS keys, SRTP state, jitter buffers, and the client playout cursor. Moving packets to a different worker mid-session breaks decryption and timing state.
Review the voice design
A tight first-audio objective depends on every handoff. A demo that answers incident 4219 once proves little: the design also needs an explicit response to jitter, interruptions, uncertain turn boundaries, and tools that can't finish inside the budget.
| Check | Pass when |
|---|---|
| Speech-end TTFA | p50 and p95 are measured on declared network slices, not inferred from one 380ms trace |
| Commit-to-audio | Isolated from endpointing so LLM, TTS, and playout regressions are visible |
| Turn detection | Local VAD mutes fast; server endpointing combines silence, stable words, and semantic completeness |
| Transcript commit | Unstable partials may prefetch reads; they may not write history, speak facts, or fire side effects |
| Slow tools | First audio is an honest acknowledgement; paging and closes wait for results or confirmation |
| Barge-in | p95 mute meets the product target, and history keeps text through confirmed playout only |
| Echo | AEC is tested on real devices, speaker volumes, and rooms |
| Transport | Browser callers use a WebRTC media path with ICE/TURN fallback; WebSockets don't impersonate that path |
| Native audio | Chosen only after measuring latency, prosody, policy control, and the same confirmed-playout cursor rule |
| Prefix cache | Stable policy and tool schemas come first; usage metadata shows cached tokens |
Test the design with one clean call, one interrupted call, and one degraded network slice. Trace a packet from microphone to model and back, calculate first-playback latency, name the bottleneck, and compare the confirmed playout cursor across cascaded and native speech-to-speech paths. Then vary endpoint policy and tool latency until you can predict which SLO moves and why.