Low-latency voice streaming: What actually determines response speed in production

Most evaluations of low latency voice streaming stop at a number on a pricing page. A vendor publishes a figure, a demo call sounds crisp in a quiet room on office wifi, and the shortlist gets made. Then the system goes live on real phone calls, over real carrier paths, at real concurrency, and the conversation starts stepping on itself. Callers interrupt and the agent keeps talking. Silences stretch just long enough that people say “hello?” into the gap.

The gap between demo and production is rarely the model. It is the parts of the path nobody scored: where audio physically enters the network, how many hops it takes before an ASR engine sees it, how long the endpointer waits before deciding a caller has finished, and what happens to the 95th percentile when 400 concurrent calls hit the same media region.

This is a criteria-led review. Eight weighted criteria, scored the way a procurement team would score them, with carrier ownership, media-plane geography and tail latency treated as first-class factors rather than footnotes.

Response speed is a procurement criterion, not a demo metric

Human turn-taking in natural conversation sits at roughly 200 milliseconds of gap between speakers. Push past about 800 milliseconds and callers start to feel the machine. Push past 1.5 seconds and they interrupt, repeat themselves, or hang up. On a consented collections reminder or a payment-failure recovery call, that behaviour shows up directly in containment and connect-to-resolution rates. Response speed is a commercial variable, not an engineering vanity metric.

The trouble is that “voice ai latency” is not one number. A production call has at least eight serial stages: PSTN ingress and codec handling, jitter buffering, packet transport to the media server, streaming egress to the ASR, endpointing decision, LLM first-token generation, TTS first-audio-chunk generation, and the return path back down the same telephony chain. Vendors publish whichever slice makes them look best. A model provider quoting 320 ms is usually measuring from end-of-speech-detected to first audio byte out, ignoring the 400 ms the endpointer waited and the 180 ms the audio spent getting to the region in the first place.

Score the whole path or you are scoring nothing.

How we scored: eight criteria for evaluating real time audio streaming for voice AI

Each criterion carries a weight reflecting how much it moves perceived response speed in live enterprise deployments, not in lab conditions.

• Measurement honesty, 15 points. Does the published latency figure cover the full mouth-to-ear path, and is the measurement method disclosed?
• Transport and protocol coverage, 10 points. WebSocket, SIP, WebRTC, gRPC, SIPREC, media forking, and what codecs survive the journey.
• Media-plane geography, 15 points. Where audio lands physically, and how many international hops sit between the caller and the model.
• Call-control depth without added delay, 12 points. Transfer, hold, DTMF, recording and consent capture available mid-stream without tearing down the media session.
• Turn-taking control, 15 points. VAD sensitivity, configurable endpointing windows, and how fast barge-in actually cuts playback.
• Model-layer flexibility, 10 points. Bring-your-own ASR, LLM and TTS versus a fixed bundled realtime API.
• Tail latency under concurrency, 15 points. P95 and P99 behaviour at production load, not median benchmarks on an idle cluster.
• Telco licensing and compliance obligations, 8 points. Whether the legally required path in your markets adds hops you cannot remove.

Criterion 1. Measurement honesty: what each vendor’s published latency number actually includes

Ask every vendor to define their measurement boundary in writing. The useful question is: from the moment the caller stops speaking, to the moment the first audible syllable leaves the phone speaker, what is the P50 and P95 on a real PSTN call originating in my primary market?

Answers cluster into three types. Some vendors measure model inference only, which typically excludes 300 to 600 ms of telephony and endpointing. Some measure their own platform boundary, from media server ingress to media server egress, which is honest but leaves carrier legs unaccounted for. A small number measure end to end on a real dialled call and will show you the trace.

Full marks go to vendors who hand over a breakdown by stage. A stage-level trace is the only artefact that lets your team find the 200 ms you can actually remove. A single aggregate number, however impressive, is not diagnosable. When Exotel references sub-300 ms voice latency, the relevant follow-up in any evaluation is the same one you should put to every vendor: which segments of the path does that cover, and how is it instrumented.

Criterion 2. Transport and protocol coverage in a voice streaming API for developers

A voice streaming API for developers needs more than one way in. WebSocket streaming of raw PCM or mu-law frames is the common denominator and works well for bidirectional model integration. SIP and SIPREC matter when you need to fork media out of an existing contact centre or PBX without re-architecting call flows. WebRTC matters for app and browser channels. gRPC shows up where teams want backpressure handling and typed contracts.

Codec handling deserves scrutiny. Every transcode adds delay and strips signal that ASR accuracy depends on. If the platform decodes G.711 to PCM, re-encodes for transport, then decodes again at the model boundary, you have paid twice for nothing. Ask what the native frame size is, whether 8 kHz audio is upsampled and where, and whether wideband survives from the carrier leg through to the ASR when the caller is on a VoLTE handset.

Bidirectional streaming is the other differentiator. Platforms that only fork audio out, with no low-delay path to inject synthesised audio back, force you to route TTS through a separate playback API. That round trip is where barge-in responsiveness quietly dies.

Criterion 3. Media-plane geography: where the audio physically lands before it reaches a model

This is the criterion that model-only comparisons ignore entirely, and it is often the largest single line in the budget.

A call originating in Mumbai, terminating on a media server in Virginia, streaming to an ASR endpoint in Oregon, and returning the same way, pays 180 to 260 ms in transit before anything intelligent has happened. No amount of model optimisation recovers that. The same call handled on a media server inside India, streaming to a model endpoint in the same region, starts the conversation with a couple of hundred milliseconds of headroom already banked.

Three questions separate serious answers from marketing ones. Where are the media servers, by city or region, for my primary calling markets? Is media anchored close to the point of origination or backhauled to a home region? And when I plug in my own model endpoint, can I pin it to the same region, or does the platform egress from a fixed location regardless?

Real time voice streaming is a geography problem before it is a software problem. Teams evaluating platforms across India, the GCC and Southeast Asia should map the physical path for each market separately, because a vendor strong in one may be backhauling in another.

Criterion 4. Call-control depth without added delay: transfer, hold, DTMF, recording and consent capture

Streaming audio to a model is the easy half. The hard half is doing everything else a phone call needs while the stream stays live.

Score each of these as available mid-stream, available with a session teardown, or unavailable:

• Warm transfer to a human agent with conversation context carried across, so the caller does not repeat themselves.
• Hold and resume without dropping the media session or restarting ASR state.
• In-band and RFC 2833 DTMF capture during an active stream, which matters for OTP entry and IVR fallbacks.
• Recording and consent capture that starts, stops and annotates without interrupting audio flow.
• Barge-in-safe prompt playback where injected audio can be cancelled immediately.

Platforms that require a teardown and re-establish for transfer add one to three seconds at the exact moment a frustrated caller is being handed to a person. In regulated outbound work such as EMI reminders or verification calls, consent capture, script adherence and audit-ready recording are non-negotiable. Any design that treats them as an afterthought bolted onto the stream will cost latency somewhere.

Criterion 5. Turn-taking control: VAD sensitivity, endpointing windows and barge-in behaviour

In most production voice agents, the endpointing silence window is the single largest contributor to perceived delay, and it has nothing to do with model speed.

If the endpointer waits 700 ms of silence before declaring the caller finished, you have spent more than twice the entire inference budget on waiting. Drop it to 300 ms and the agent interrupts people mid-thought. The right window varies by call type, by language, and by whether the caller is reading a number aloud or telling a story.

Score platforms on whether these are exposed as tunable parameters rather than fixed defaults:

• VAD aggressiveness and separate thresholds for noisy versus clean channels.
• Endpointing windows configurable per prompt or per conversation state.
• Barge-in cut-through time, measured from the caller’s first syllable to playback stopping. Anything over 150 ms feels rude.
• Backchannel handling, so an “mm-hmm” does not trigger a full turn.
• Noise and accent resilience, tested on the actual channel conditions of your callers, including handset calls from moving vehicles and street-side environments.

A vendor that gives you a fixed 800 ms endpoint and calls it a feature has removed your best lever for hitting sub 300 ms voice latency on the parts of the path you control.

Criterion 6. Model-layer flexibility: bring-your-own ASR, LLM and TTS versus a bundled realtime API

Bundled realtime APIs, where speech-to-speech happens inside one provider’s model, are genuinely fast and genuinely simple. They also lock the entire linguistic layer to one vendor’s roadmap, which is a problem if your callers speak Hinglish, code-switch mid-sentence, or need Arabic dialect coverage the model handles unevenly.

Component stacks, where you choose ASR, LLM and TTS separately, cost you a few tens of milliseconds in orchestration and buy you the ability to swap a weak component without replatforming. For BFSI workloads where number recognition accuracy in Indian English determines whether a payment flow completes, that swap capability is worth more than the milliseconds.

Score for optionality: can you point the stream at your own endpoint, run a bundled path where it fits, and mix approaches across use cases on the same infrastructure? Locked-in either direction loses points.

Criterion 7. Tail latency and behaviour under concurrency, not median benchmarks

Median latency is close to meaningless for customer experience. A caller does not experience your P50. They experience the one turn in twenty where the response took 2.4 seconds, and they form their opinion of the whole system there.

Insist on P95 and P99 figures, gathered at your expected peak concurrency, in your primary region. Then ask what happens at the edges:

• Cold-start behaviour when a new model container spins up mid-campaign.
• Queueing under burst, for example when a dialer campaign launches and 800 calls connect within 30 seconds.
• Degradation mode when the model endpoint is slow. Does the platform inject a filler, hold silence, or drop the call?
• Jitter and packet loss handling on mobile-originated calls, where the last mile is the least controllable part of the path.

Ask about dropped-call behaviour explicitly. A platform designed to minimise silent media failure, with telecom-grade fallbacks, protects tail experience in a way that no model optimisation can.

Criterion 8. Telco licensing and compliance obligations that constrain the path in India, the GCC and SEA

Regulation shapes the physical path, which means it shapes latency. In India, licensed connectivity and the requirement to originate and terminate through authorised telecom infrastructure determines where media can be anchored. In the UAE, number provisioning and call handling sit under Consumer Protection Regulation and Standards and Consumer Protection Standards oversight. Indonesia and the Philippines bring their own supervision through OJK and BSP for financial services outbound, alongside expectations around consent, recording and script adherence.

A platform that holds carrier relationships and licensed number infrastructure in your markets can keep the media path short while working within the licensed routes those markets require. A platform that does not will either route around the problem, adding hops, or partner with a local carrier, adding a hop and a support boundary. Score for licensed presence in each market you actually operate in, and for whether consent capture, audit-ready recording and script-adherence scoring are native to the same stack that carries the audio. Regulatory alignment described here is capability, not legal advice, and your compliance team should validate against current rules.

Category scorecard: how the main types of voice streaming platforms for AI agents perform against the criteria

Grouping by architecture is more useful than grouping by brand, because the architecture predicts the failure mode.

Bundled realtime model APIs. Strong on raw inference speed and developer ergonomics. Lighter on media-plane geography in Asia and the Gulf, thinner on call control, and typically silent on telco licensing because they do not carry calls. Good scores on criteria 1 and 6 if you only count their own boundary; lower scores on 3, 4 and 8.

General-purpose CPaaS with media forking. Solid transport coverage, mature call control, published APIs developers like. Media geography varies sharply by market, and the AI layer is usually someone else’s, which means the ASR-to-TTS loop leaves the platform and comes back. Reasonable on 2 and 4, mixed on 3 and 7.

Voice AI startups on third-party telephony. Excellent turn-taking tuning, often the best barge-in behaviour in the category, because that is their whole product. The limitation is structural: they do not own the carrier leg, so when latency spikes or calls drop, the diagnosis crosses a vendor boundary. High on 5 and 6, more exposed on 3, 7 and 8.

On-prem SBC and telephony stacks with bolted-on AI. Full control of the media path and the shortest possible hops inside a data centre. Slower to change, heavier to operate, and the model layer usually lags. Strong on 3, lower on 6 and on time-to-deploy.

Unified telco-plus-AI platforms. Own the carrier layer and the AI layer on one architecture. The advantage concentrates in criteria 3, 4, 7 and 8, which happen to carry 50 of the 100 points, because those are the criteria that require owning infrastructure rather than integrating it.

How a unified telco-plus-AI stack scores: reading Exotel against the same eight criteria

Applying the scorecard to Exotel’s own approach, the pattern follows the category logic above.

On media-plane geography, Exotel operates across 11 telco circles in India with licensed local number infrastructure in the UAE under TRA and CBUAE oversight, which means media can be anchored close to origination rather than backhauled. The AgentStream voice-streaming infrastructure and the StreamKit Cloud Connector are built to hand audio to model endpoints without the extra international hop that dominates cross-region deployments. Exotel publishes sub-300 ms voice latency alongside 99.99% platform uptime and a zero-dropped-call design. The honest framing is that those are platform figures reported by Exotel on its own infrastructure, not a guarantee for every route and every model you attach.

Call-control depth scores well because voice, contact centre and AI sit on one architecture. Warm handoff from an AI voice agent to a human carries full conversation context, one agent can supervise several AI conversations at once, and consent capture with audit-ready recording is part of the same stack rather than a separate system stitched in. For consented collections, verification and BFSI outbound, script-adherence scoring and Conversation Quality Analysis run across all interactions rather than a sampled subset, which supports alignment with frameworks such as RBI FPC, OJK, BSP and TRA and CBUAE expectations without standing in for your own compliance review.

Turn-taking is handled through noise-resilient ASR and barge-in and interruption handling built into the Voicebot layer. Multilingual coverage spans English, Hindi, Hinglish and Arabic among others, which matters because endpointing behaviour differs across languages and code-switching callers break naive VAD.

Model-layer flexibility is where any bundled platform has to be examined case by case. The streaming infrastructure exists precisely so that audio can reach the models you want it to reach, and the persistent context layer keeps customer state across bots, agents and channels rather than resetting at each hop.

The honest trade-off of any unified stack is the same as its strength: consolidation. If your organisation deliberately wants best-of-breed at every layer with independent replacement cycles, a single-vendor architecture is a trade-off you should price explicitly, against the vendor finger-pointing and broken context that fragmented stacks tend to produce.

Shortlist questions to put in the RFP

Take these verbatim into the vendor questionnaire. Vague answers are answers.

• State your P50, P95 and P99 mouth-to-ear latency on a PSTN-originated call in each of our primary markets, and describe the measurement method and boundary.
• List the cities or regions where media is anchored for our markets, and confirm whether audio is backhauled to any other region.
• Provide a stage-level latency breakdown covering ingress, transport, ASR, endpointing, LLM first token, TTS first chunk and egress.
• Confirm which endpointing and VAD parameters are configurable at runtime, per conversation state, and give the measured barge-in cut-through time in milliseconds.
• Describe mid-stream call control: warm transfer with context, hold and resume, DTMF capture, and recording start and stop, and state which of these require a session teardown.
• Document behaviour under burst concurrency, including cold-start handling, queueing, and what the caller hears when the model endpoint is slow.
• Confirm licensed telecom presence and number provisioning in every market we operate in, and name the regulatory frameworks you align to, including consent and recording obligations.
• Specify how we attach our own ASR, LLM or TTS endpoints, what latency that adds, and whether those endpoints can be pinned to the same region as the media server.
• Explain the support and escalation path when latency degrades, and identify who diagnoses the carrier leg.
• Provide reference architecture and results from a deployment at comparable volume and in a comparable regulatory environment.

FAQs