Realtime Voice AI Systems

Key takeaways

• 01

  Time-to-first-audio around 800ms (and turns under ~1.2s) is the product constraint; quality work after that is wasted if the call already feels laggy.

• 02

  Barge-in is mandatory: stop TTS within ~100ms of detected caller speech, or the agent talks over the human.

• 03

  Turn-taking is a state machine (listening, thinking, speaking, barged-in), not a prompt instruction.

• 04

  Stream every hop — STT partials, LLM tokens, TTS chunks — so you never wait for a full sentence before the caller hears sound.

• 05

  Evaluate latency percentiles, barge-in success, word error on your vocabulary, and task completion on recorded calls, not demo scripts.

Five pieces sit on the media path. A telephony or WebRTC plane (SIP trunk, CPaaS, or in-app WebRTC) carries bidirectional audio with a jitter buffer you control. Streaming speech-to-text emits partial hypotheses and a final, with word timings and a custom vocabulary for product names. A turn-taking controller uses voice activity detection, endpointing, and barge-in to decide whose turn it is. A low-latency LLM produces short spoken turns from a dialogue state object, not a novel. Streaming TTS starts audible speech on the first sentence or less, and can be killed mid-byte.

This page is the architecture of that loop. Domain logic — lead qualification schemas, appointment booking, IVR replacement — lives above it. If the media path is slow or barge-in is missing, no dialogue design will save the call.

Audio frames in, audio frames out. Partial STT updates the working transcript while the caller is still speaking. Endpointing (silence, punctuation, or a learned end-of-turn model) flips the state to thinking. The LLM is already warm: prompt cache for the system prompt and tools, first token on a fast model. TTS begins on the first clause, not the full answer. The sum you actually feel is endpointing delay + STT finalization + LLM time-to-first-token + TTS time-to-first-byte + playout buffer. Target ~800ms from end-of-caller-speech to first agent audio; keep full turns under ~1.2s. Above ~2s, overlap and hang-ups dominate.

Do not wait for a “final” transcript if a confident partial plus endpointing is enough to start thinking. Do not let the LLM write 80-word answers; spoken turns belong under ~35 words and should end on a question or a next step. Prefetch TTS voices and keep a hot inference path; cold starts blow the budget by themselves.

• Stream STT partials; do not block on finals to start planning
• Cap spoken turns; long answers destroy barge-in and latency
• First TTS audio ~800ms after caller endpoint
• Kill playback within ~100ms of barge-in
• Keep the media path in one region as the LLM

Turn-taking is a state machine: listening, endpointing, thinking, speaking, barged-in, silent-timeout, hold. Prompts cannot enforce it. Voice activity detection that is too aggressive chops the caller; too timid leaves dead air. Endpointing should be adaptive — shorter after a yes/no, longer after “my account number is…”. Echo cancellation and residual TTS in the mic are what cause the agent to interrupt itself; that is an audio problem, not an LLM problem.

Barge-in: on caller speech above threshold during speaking, stop the TTS stream, discard queued audio, keep the overlapping transcript, and re-enter thinking with the new content. If stop-to-silence is slower than ~100ms, callers hear the agent finish a clause they already talked over and assume they were ignored. False barge-in from background noise needs a second gate (energy + ASR partial confidence), not a disabled barge-in flag.

Latency is the first outage customers notice. A 300ms STT that waits for punctuation, an LLM that does not stream, a TTS that synthesizes the whole paragraph, or a region hop between telephony and model will each eat the 800ms budget. The second outage is talk-over: missing barge-in, slow kill, or echo that looks like caller speech. The third is transcript error on the tokens that matter — names, amounts, account ids — which the LLM then treats as facts.

Operational failures: one-way audio from a SIP re-invite, clock drift on recordings, tool calls that block the thinking state without a spoken filler, and compliance gaps (no AI disclosure, recording without consent, missing DNC). Tooling must be async or fast; a 2s CRM lookup in the thinking state is a dead-air bug. If a tool will take longer, speak a hold phrase and keep barge-in live.

• Budget missed because a hop is batched instead of streamed
• No barge-in, or barge-in slower than ~100ms
• Echo causing self-interruption
• STT errors on names, numbers, and product terms
• Blocking tool calls that create dead air

Measure the pipeline, then the dialogue. Pipeline: time-to-first-audio p50/p95, barge-in kill time, false barge-in rate, STT word error on a domain vocabulary set, TTS time-to-first-byte, one-way-audio incidents. Dialogue: task completion on a golden set of recorded or simulated calls, interruption handling, disclosure spoken in the first turn, and correct writes to the system of record. A judge on transcript quality that ignores latency will green-light an un-callable agent.

Replay production recordings through the STT and dialogue core as a nightly suite. Add adversarial audio: background TV, dual talk, accented speech, DTMF. Track hang-up time as a product metric; it often moves before your task-success number does.

Do not put voice in front of a workflow that still needs a 3s LLM, a blocking database, or a human approval mid-sentence — the architecture will not hide that. Do not use voice when the legal posture on recording, disclosure, or outbound calling is unresolved. Do not use it as a cheaper IVR tree if DTMF plus a short audio prompt already completes the task; the failure modes above are not free.

Use realtime voice when the conversation is the interface: inbound qualification, support deflection with warm transfer, appointment changes, status that callers will not type. Pair it with a typed state object (the lead, the ticket, the booking) so the call is filling a schema, not performing a script. If you cannot stream STT, LLM, and TTS today, ship a callback or chat path instead of a laggy phone agent.

Frequently asked questions

Keep reading