Key takeaways
- Five stages sit between your caller falling silent and the first word back, and endpointing costs the least to change.
- Published latency figures usually describe a single stage. Measured end-to-end, real systems come in far slower than the numbers vendors quote.
- Callers read a pause as reluctance. In conversation research, gaps past 600 milliseconds start producing that inference.
- An agent can answer fast and still talk over the caller who interrupted it.
- Latency climbs on longer calls and under higher throughput.
A caller finishes a sentence. The agent hears silence, waits to be sure the sentence is over, and only then starts working out a reply. People never do this. They work out where a sentence is going while it is still being spoken, and answer the moment it stops.
What voice AI latency really means on a phone call
Voice AI latency is the time between a caller finishing their sentence and hearing the first word back.
The benchmark your callers apply comes from ordinary conversation. Researchers measuring question and answer exchanges across 10 languages found the most common gap between speakers fell between 0 and 200 milliseconds. Japanese speakers averaged 7 milliseconds. Danish speakers, the slowest in the sample, averaged 468. Timing runs tighter on the phone than face-to-face, so a caller is applying the stricter version of that expectation.
Speakers reach those numbers by predicting. Producing the words takes over 600 milliseconds, and the gap between turns runs near 200, so a reply gets built while the other person is still talking. A system waiting on a silence threshold has nothing equivalent, and part of its delay is set in a config file.
Time to first token marks when a language model produces its first word internally, and the caller hears nothing at that moment. Speech synthesis has not started, and the audio has not crossed the telephony network.
What a caller hears when your agent goes quiet
Callers reach for the phone when self-service has failed them. Gartner surveyed 5,728 customers and found self-service fully resolves only 14% of issues. In a McKinsey survey of 3,500 consumers, live phone conversations ranked among the most preferred ways to reach a company for help, including among Gen Z.
Two things can go wrong once that call connects, and they pull in opposite directions:
- The moment a caller decides the line is dead
A pause is not neutral in conversation. Corpus analysis shows gaps of 700 milliseconds or more are associated with dispreferred responses, and experiments show gaps of 600 milliseconds or longer produce the same inference.
Your caller applies that reading to your agent, which they cannot see working.
- What happens when you tighten the threshold
Tightening the silence threshold to close the pause produces a different failure. The agent starts speaking while the caller is still mid-sentence.
Researchers analyzing recorded human-agent conversation named two separate faults, one being an agent too slow to start speaking and the other being an agent too slow to stop once someone talks over it. Callers meet both on the same call, and neither setting fixes the other.
Where the delay comes from in a voice call
A cascade runs its stages one after another. Nothing reaches the model until the transcript is final, and synthesis waits on the first words it produces. Researchers building voice agents describe the cost plainly, noting that these pipelines execute strictly in sequence, unlike a conversation where a listener starts thinking before the speaker finishes.
Deciding the caller has finished is a judgment called endpointing, and it costs several hundred milliseconds while taking the least amount of work to change. Voice Activity Detection (VAD) classifies each audio frame as speech or silence, and endpointing applies a rule on top of that. A common approach waits for a fixed period of silence, often 500 milliseconds or a full second, before the system takes its turn. That time goes before any processing starts.
That chain has five stages, and two of them wait on the stage before:
Newer approaches add a turn-detection model that waits longer when the words suggest the caller has not finished, and some pair that with the running transcript. Benchmarked against silence alone, they cut false triggers sharply, and the fastest of them still answers later.
Transport falls outside your software, and it is already tight. The ITU recommendation for voice networks puts a well-routed regional call under 150 milliseconds mouth to ear, and a call crossing 10,000 kilometers at roughly 225. Jitter buffering adds about half of the variation it is built to absorb. A cascade spends several hundred milliseconds more on top of a path engineered to that standard.
Why a demo call hides what happens under load
A demo runs one conversation against a system doing nothing else. Your callers arrive together, and the capacity behind the agent is often shared with everyone else's callers.
Researchers replayed real queries at eleven concurrency levels up to 50 simultaneous users. On shared capacity, median response time more than doubled. The slowest five percent of requests stretched half again beyond that median. Reserved capacity held up until its allocation saturated, then degraded too.
The finding that matters for a phone line is that the tail moved before the median did, and moved further. A demo never shows you the tail.
So a quoted latency number needs two questions put to it. How many calls were running when it was measured, and in what capacity?
Why the tenth minute feels slower than the first
A model holds nothing between turns, so the whole conversation goes back in with every new question. What arrives at minute ten is far longer than what arrived at minute one, and the wait before the first word grows with it.
Researchers measuring a full-history setup found total turn time growing rapidly with dialogue length, more than doubling between a ten-turn history and a thirty-turn one.
On a booking call, the longest exchanges come last. The caller has already given a date, changed it, and spelled out a number by the time they ask the question that decides whether they book.
How to reduce latency without cutting callers off
Two of the changes below shorten the pause itself. The third handles the interruption problem that a faster agent makes visible:
- Move endpointing off the network
Cloud recognition decides that a caller has stopped by sending audio away and waiting for an answer, so network time lands inside the pause. A local model that predicts when someone is finishing removes that round trip.
Researchers tested both in a shopping mall over two days. Average response time fell from 2.14 seconds to 1.15, and on the turns the prediction model handled alone, it reached 0.71 seconds. Users answered faster themselves and rated the conversation significantly smoother.
Ask your platform whether endpointing runs locally or waits on a cloud response.
- Stream every stage
Each stage can start before the previous one has finished. Recognition emits partial text while the caller talks, and synthesis begins on the first words available.
Confirm that your recognizer and your synthesis each work this way. A single stage that waits for a complete result holds up everything behind it.
- Give interruptions their own handling
Faster endpointing does nothing for the agent that keeps talking after a caller cuts in. Something has to watch the line as the agent speaks, and act on what it hears.
In one research system, a separate stream monitors the caller throughout, and when it detects an interruption, playback is halted, and the audio still queued behind it is thrown away. Without that second part, the agent keeps speaking from a buffer the caller has already talked over.
Call your own agent and interrupt it. Anything you notice as a caller is the queued audio still playing out.
How to measure latency on your own calls
Ask what a published number measures. One study timed four voice systems the same way, from the end of the user's audio to the first audio chunk reaching the client. Every one of them landed between two and four seconds at the median. Figures in the low hundreds of milliseconds start the clock somewhere later.
Log four timestamps on your own calls:
- When your caller stops speaking
- When their transcript goes final
- When your model emits its first token
- When audio reaches the line
Time from the first to the last is what a caller sits through, and what falls between them tells you which stage to work on.
Report the median alongside the 95th percentile. The first tells you about an ordinary exchange. The second catches the one where somebody starts talking into silence.
How Phonely holds its response time steady
Phonely's Alma model reports a median of 182 milliseconds to first token against 490 for GPT-4.1, measured on 200 sequential requests from a single region. That covers the model stage. Synthesis and the telephony network come after it.
Alma's median-to-p99 gap on time to first token is 24 milliseconds, whereas GPT-4.1 goes from 490 to 2,018. Phonely credits dedicated serving rather than the model, and the test ran one request at a time with no concurrency.
On long calls, the benchmark is direct about what is unsolved. It names how conversation history is tracked and fed to the model as the remaining headroom, and says it is shared across every model tested. The other four stages are yours to instrument.
Frequently asked questions
- Should I use a speech-to-speech model instead of a pipeline?
Speech-to-speech models answer sooner and handle interruptions natively. Researchers building them also note that they produce semantically weaker outputs than a pipeline that routes text through a language model. On a call that books an appointment or takes account details, getting the details right is the harder requirement.
- Does looking something up mid-call slow the answer down?
Yes, and it is the easiest cost to miss. A production vector database query adds 50 to 300 milliseconds of network round trip, which on its own can consume the whole budget for a natural reply. Researchers cut that to under a millisecond by predicting the likely next question and fetching the answer before the caller asks it.
- Can I keep long calls fast by trimming the conversation history?
Partly. Cutting to the most recent turns discards earlier information the agent may need later, and summarizing the history can go stale as the call moves on.
