Voice AI FAQ
Voice AI is a latency and turn-taking problem first, and a language-model problem second — plus the legal duty to say so on the call.
Voice AI turns spoken audio into text, lets a model decide the next step, and speaks back, usually over the phone or in an app, with a turn budget around one second. It is not a menu tree, and it is not a chatbot with a speaker bolted on. If you cannot interrupt it, disclose it, or hand the call to a person, it is not ready for production.
What voice AI is — and is not
If you only need one number: keep the agent’s turn under about 1.2 seconds, disclose it, and give the caller a way out.
What is voice AI?
Software that listens, decides, and speaks: streaming speech-to-text, a model that chooses the next sentence or tool call, and streaming text-to-speech, usually on a phone call or in-app audio. The hard parts are turn-taking, latency, interruption, and writing a structured result to a system of record. A recorded IVR prompt played by a neural voice is not voice AI in this sense. Neither is a chat model that you read aloud.
How is a voice agent different from IVR?
IVR follows a flowchart: press 1, say a keyword, hang in a loop. A voice agent fills a schema from messy speech, handles two answers in one sentence, and can look up or change a record under scoped permissions. That flexibility is why it needs evals and a human transfer. Keep IVR for deterministic, high-volume menus. Use an agent when the call needs judgement over language.
What latency does a voice agent need?
End-to-end, from the caller finishing a sentence to the agent starting to speak, should sit under about 1.2 seconds; past two seconds, talk-over and hang-ups climb fast. That budget is split across STT endpointing, the model’s first tokens, and TTS time-to-first-byte. A smarter model that adds a second of thinking usually loses to a faster one on the phone. We design the pipeline to the budget before we pick the model.
What is barge-in on a voice agent?
Barge-in is stopping the agent’s playback as soon as the caller starts talking, typically within about 100ms, and treating the new speech as the next turn. Without it, the agent talks over the customer and the customer hangs up. It is a telephony and TTS-control feature, not a prompt. If a vendor demo cannot interrupt itself, the demo is not a product.
Do we have to disclose that the caller is speaking to AI?
In a growing list of US states and under EU transparency rules, yes — and commercially you should even where the statute is quieter. Put it in the first sentence. Hiding it fails legally in some places and fails the call everywhere when the caller notices. Disclosure is not GDPR consent and not recording consent; those are separate. Your legal team owns the wording; we put the sentence in the opening turn.
Can voice AI replace our contact centre?
Not in month one, and usually not as a total replacement. It can take high-volume, well-bounded calls — status, qualification, appointment windows — and warm-transfer the rest. Teams that try to remove humans from identity, billing disputes, or clinical content put them back. Measure containment on targeted intents, not a fantasy of zero headcount. We will say if the first workflow is too wide.
Which part matters most — STT, TTS, or the LLM?
The pipeline. Bad endpointing or slow TTS will kill a strong model; a beautiful voice will not save a three-second wait. STT vocabulary (product names, SKUs) and TTS barge-in are where most first deployments break. The model matters once turns already feel like a conversation. We pick each piece against the latency budget and the language of your callers, model-agnostic on the LLM.
How a voice agent gets built
The first production voice agent is a schema, a telephony path, and a transfer — not a 40-prompt script.
How do you keep turn latency under about a second?
Stream everything: partial STT, first-token model output, sentence-level TTS. Skip reasoning models on the hot path unless the extra seconds are worth losing the caller. Cache greetings and frequent answers. Measure p50/p95 on real calls, not on a laptop demo. If the budget is blown, we cut model size or thinking before we add another retrieval hop.
Can a voice agent write to our CRM?
Yes, and that is usually the point. Every call should leave a transcript, a structured object, a score, and a next step on the record. Write actions use the credentials you issue and can sit behind a confirmation or a confidence gate. Connecting a listed CRM integration does not cost extra. If there is no API, we will say so before week two.
How do you handle accents and noisy phone lines?
Custom vocabulary, domain language packs where the STT vendor has them, and an eval set that actually includes your callers — not studio audio. Noise and overlapping speech still produce errors; the agent should confirm critical slots (phone numbers, amounts) rather than guess. We will not claim a percent accuracy we have not measured on your traffic.
What do you need from our team to build a voice agent?
A telephony owner who can provision numbers and media streams, one process owner for the call outcome, one engineer with CRM or helpdesk access, and a weekly 45-minute review. Legal sign-off on recording, disclosure, and do-not-call is required before live traffic. That is the input set; we do not need a standing steering committee.
How do you stop the agent talking over the caller?
Barge-in on the media path: detect speech, kill playback, discard the rest of the generated audio, and start a new turn. Keep agent sentences short so there is less to talk over. Endpointing that waits too long feels sluggish; endpointing that cuts too early clips the caller. We tune that on real calls in shadow or limited rollout, not in a script read-through.
How is a voice agent priced?
A fixed-scope implementation fee plus the platform subscription, quoted before week one. Telephony, STT, TTS, and LLM usage are paid by you to those providers at your rates — no token or minute markup from us. Minutes and media are the cost line people underestimate; we will put a forecast in the quote rather than hiding it.
Is four weeks realistic for voice, or is that chat-only?
Four weeks is the standard for one bounded workflow when telephony access and legal sign-off show up in week one: discovery, environments, shadow on real calls, handover. If numbers, recording consent, or the CRM API slip, the calendar slips. We will not pretend a contact-centre replacement is a four-week job. Most teams hear the first agent on real audio in week three.
On the phone, in production
Production is recordings you are allowed to keep, a transfer that actually rings, and evals on calls — not on chat logs.
Where does the audio pipeline run?
In your telephony account and your cloud. Media streams, recordings, and transcripts stay in your perimeter; we do not park call audio on ReinforcedX disks. STT/TTS vendors are subprocessors you contract. For regulated work we keep the model and the logs in your VPC where the vendors allow it. Consumer “AI phone” apps are not this architecture.
How do you store recordings and transcripts legally?
Only with the consent and retention rules your counsel sets. We implement recording flags, two-party or one-party notice, deletion jobs, and access control on audio and text. Defaulting to “keep everything forever for training” is how you fail GDPR and sector rules. Client data is not used to train shared models. Your DPO owns the schedule; we implement it.
What happens when the caller asks something off-script?
The agent answers from retrieved policy if the question is in-domain, confirms it does not know if retrieval is empty, and transfers when the intent is high-stakes or out of scope. A rigid script that refuses to leave the tree is IVR. An agent that improvises medical or legal advice is a defect. The schema plus a transfer policy is the control; we evaluate missed-transfer as a first-class metric.
Can we warm-transfer a live call to a human?
Yes, and you should build it before polish. The agent whispers a brief — reason, slots filled, sentiment — and the human takes the same call. Hot-lead transfer beats a callback form; angry-customer transfer beats a survey. If your telephony stack cannot conference or whisper, we will say so in week one rather than promising a feature the carrier does not offer.
How do you evaluate a voice agent?
A golden set of real calls with labelled outcomes, plus latency percentiles, barge-in success, slot accuracy, and missed-transfer rate. Chat evals are not a substitute; people interrupt and audio drops. CI gates fail a release when those numbers drop. Online sampling of live calls catches a silent STT or model update. Humans still listen to a slice — that is where the 98.7% QA pass rate is earned.
What does handover include for a voice agent?
A running number or in-app session in your environment, the schema, prompts, eval suite, latency dashboards, a runbook, and 30 days of on-call. Recordings and transcripts stay in your stores. Your engineers sat in the weekly review, so handover is operations, not a PDF. Most teams keep a quarterly eval pass; that is optional.
Where voice pays off — and where it does not
The pipeline travels. The disclosure, recording, and transfer rules do not.
Is voice AI only for sales qualification?
No. Qualification is a common first workflow because the success metric is clear. The same pipeline serves appointment booking, order status, collections reminders, and staff-admin lines. What changes is the schema, the transfer policy, and the compliance pack. For the inbound sales build itself, use the lead-qualification how-to rather than this FAQ as the playbook.
Can we use voice agents in healthcare or finance?
For admin intents — scheduling, benefits navigation, card-activation, balance-with-auth — yes, with HIPAA-aware or sector controls, recording rules, and a ban on improvised medical or financial advice. Clinical diagnosis and unauthorised money movement stay off the agent. Those programmes usually run longer than four weeks. Missed-escalation rate is a target, not a hope.
Does multilingual voice AI actually work?
It works when STT, the model, and TTS all support the language, and when your eval set includes it. Mixing languages mid-sentence still breaks many stacks. We will not claim “all languages” from a vendor brochure. Scope the first two languages you will measure, then add. A bad second language on the same number can damage the first.
Should we build on the phone or in-app?
Phone if that is how customers already reach you and you can provision numbers. In-app if you control the client, want better audio, and can skip PSTN quirks. Many teams do inbound PSTN first because the volume is already there. We will not force a WebRTC rewrite if SIP is what you have in week one.
When should we not use voice AI?
When the call is fully deterministic (keep IVR), when you cannot define a success metric, when you cannot disclose or record legally, or when a wrong spoken commitment is irreversible and you have no human on the line. Also when you cannot meet a ~1.2s turn — a slow “smart” agent is worse than a short menu. We will say no rather than ship a three-second oracle.
A production voice agent is streaming speech-to-text, a low-latency model, streaming text-to-speech, barge-in, and a structured object you write to the CRM or helpdesk. IVR follows a tree; a voice agent fills a schema and takes scoped actions. Sub-1.2s turns matter more than a clever prompt. Disclose that the caller is speaking to an AI. ReinforcedX builds the pipeline in your telephony and cloud, with evals on real calls. The lead-qualification how-to covers that one workflow in depth; this FAQ is the category.
Keep reading
Ready to refine
your workflow?
Share your current process. We’ll help you identify what can be automated — and where efficiency can be reclaimed.