Your Voice AI demo sounded sharp in the browser. On a real mobile number, callers talk over the bot or say it sounds robotic.
Usually the model is fine. The path between the phone call and your bot is where the time goes — and some of what you lose there, nobody can give back.
Where the time in a turn actually goes
Teams optimise the wrong segment because they never break the turn down. Before tuning anything, know which part you are paying for:
| Segment | Who owns it | What you can do about it |
|---|---|---|
| End-of-utterance detection | Your VAD and endpointing config | The biggest lever, and usually the one nobody has tuned |
| Phone network leg | The carrier and last mile | Fixed. Design around it |
| Media path between the call and your socket | Us | 20 ms frames, described below |
| Model inference | Your STT/LLM/TTS or speech-to-speech stack | Usually the largest controllable cost |
| Audio back to the caller | Us, then the carrier | Mirrors the inbound path |
| Your own buffering and queueing | Your bot process | Often the part nobody measured |
We are not going to put a target number on a “natural” conversational turn. Figures get quoted for that, but we have not measured it ourselves, so we would only be repeating someone else’s number. Measure your own end-to-end turn time with your real model and use that as your budget.
What our 20 ms actually is
On the phone side we move audio in 320-byte chunks of 16-bit PCM at 8 kHz — exactly 20 ms of audio per frame — ticker-paced, in both directions.
Be precise about what that covers, because an unqualified latency number leads to the wrong budget.
It is one-way, forward path: the call to your socket. It is not a round trip.
- A conversational round trip crosses the media path twice — call to bot, then bot back to caller. Each direction is paced separately. Budget it twice, not once.
- It excludes everything outside that path: carrier transit and last mile, the internet hop from us to your endpoint, your model’s inference, and your own buffering or GC pauses.
Many platforms publish no streaming figure at all, so there is often nothing to compare against. We are not going to quote competitors’ numbers next to ours — different vendors measure at different boundaries and in different directions, and a ratio between two differently-measured numbers is not a comparison. If you want to compare, measure both yourself at the same boundary.
Run your bot in the region that serves your traffic, so the network hop does not consume the headroom you just paid for.
Sample rates: what we do, and what it does not fix
This is the part most often overstated, including by us, so here it is plainly.
The phone leg is 8 kHz. Telephony audio reaches us at 8 kHz, and that is what the PSTN carries. You can configure your stream for 8, 16 or 24 kHz. If you ask for 16 or 24 kHz, we upsample from 8 kHz before sending it to you. On the way back, whatever rate your bot sends is downsampled to 8 kHz before it reaches the caller.
What follows:
- Upsampling does not add acoustic information. A 24 kHz stream from us contains what the 8 kHz phone leg captured, interpolated. It is not wideband audio.
- Ask for 16 or 24 kHz because your model requires that input format, not because it improves fidelity. Several realtime stacks accept only wideband input; this is how you feed them without writing conversion code.
- Narrowband is a limit of the phone network, not something we can engineer away. At 8 kHz the usable band tops out around 4 kHz, and consonant detail above that — sibilants, aspiration, some retroflex contrasts that matter in Indic languages — is not in the signal. Any vendor putting a bot on a PSTN call works with the same constraint. Plan your ASR expectations and your eval sets around narrowband input.
The real benefit is narrower than “better audio”, and still worth having: you do not ship a resampler in every worker, conversion happens identically on every call, and you can point a wideband-only model at a phone call without writing that code yourself.
Two costs to know about: resampling uses CPU, which matters at high concurrency, and if a resample fails we send the original 8 kHz audio rather than dropping the frame. If your bot assumes it always receives the rate it asked for, handle the case where it does not.
Stereo recording is available with the caller and bot on separate tracks, which is what makes interruption behaviour scoreable in QA rather than just transcripts.
Barge-in, marks, and DTMF
The bidirectional stream gives you three primitives:
- Barge-in(Clear) — the caller speaks, the bot yields
- Playback marks(Mark) — you learn when your audio actually finished playing
- DTMF(DTMF) on the same session — menus, OTP, consent, without a second channel
The marks matter more than teams expect. Without them you know what you handed us, not what the caller heard. After an interrupt that distinction decides whether you repeat, resume, or move on. Guess wrong and the bot either repeats itself or silently drops something the caller never received.
One rule follows: do not block the socket read loop on a slow model call. A synchronous LLM or TTS call inside the read loop turns inference latency into missed interrupts, which callers experience as the bot talking over them. Buffer, cancel, or hand off to a worker.
The stream is not a durable queue
When your bot or the path slows, we prefer keeping the conversation current over accumulating a backlog of stale audio. Interrupt events preempt audio already in flight.
That is correct for a conversation and wrong as an assumption in your code. So:
- Make TTS generation cancellable, and cancel it when the caller interrupts
- Track what the caller actually heard via marks, not what you enqueued
- Never treat “I sent it” as “they heard it” in your conversation state
- If your bot is falling behind, shorten its responses rather than buffering more of them
The first second of the call
Silence while a bot warms up reads as badly as latency, and it lands before the caller has any reason to stay on the line.
How you fill it depends on which entry you use — the second post covers choosing one.
| If you use | How you avoid dead air |
|---|---|
| ExoML (code-owned calls) | On answer, start the stream and a short say/play together; stop the say/play when the stream is ready |
| Flow (App Bazaar) | Cached greeting before actual media flows on wss |
| Connect Voice AI API | Outbound answer goes straight to the bot; your bot owns first audio. There is no Exotel-side greeting on this path |
On ExoML the parallel greeting is a documented sequence: start the stream and a short say/play on the answered event, then stop the say/play once the stream signals ready.
How fast the caller hears that filler is mostly your deployment, not ours. It is the answered event reaching your server, plus your handler’s decision time, plus your TTS starting to render. Our frame path contributes very little. Measure it in your environment, and if the filler is late, look at where your handler runs and how cold it is before anything else.
What your endpoint has to survive
Most first-week failures are environment assumptions rather than protocol bugs:
| Requirement | Why it bites |
|---|---|
| Idle timeout above your maximum call length | Load balancers and proxies commonly idle out well below call duration. A quiet stretch mid-call looks like an idle connection and gets reaped |
| A warm accept path | We connect at call time, not in advance. Cold-start latency lands in the caller’s first seconds, the least forgiving part of the call |
| Valid TLS and an allow-list that includes us | Silent handshake failures look exactly like a broken bot and cost hours to separate |
| A non-blocking read loop | See above. This is the single most common cause of talk-over |
Endpoint auth is IP allow-listing, Basic auth, or both. Decide before your first load test, not after your first security review.
One more, specific to the WebSocket path: your deploy strategy is an audio quality decision. A rolling restart that drops sockets mid-call is a dropped conversation, not a retried request. Drain connections before terminating pods, and rehearse it at low volume.
Checklist
- Break your turn budget down before optimising anything.
- Measure end-to-end turn time with your real model, not a stub TTS.
- Run the bot in the region serving your traffic.
- Set the sample rate your model needs as input, and expect narrowband content regardless.
- Handle the case where you receive 8 kHz after asking for more.
- Tune endpointing. It usually outweighs every other latency fix available to you.
- Implement barge-in and consume playback marks on duplex bots.
- Never block the socket on long model calls.
- Make TTS cancellable.
- Set idle timeouts above your maximum call length and keep the accept path warm.
- Drain connections on deploy.
- Keep a stereo eval set recorded at your production sample rate, so audio and model regressions stay separable.
In short
Voice AI on the phone fails when the streaming path was not built for agents that talk back in real time. We move audio in 20 ms frames, hand your model the input format it needs, and give you the signals to handle interruptions properly.
We do not make a narrowband phone call sound like a studio recording. Nobody does — it is not in the signal.
Next: choosing an integration path, and running this in production in India.