Building AI Assist: Engineering a Real-Time Agent Copilot

AI Assist listens to a live call and puts the transcript, a suggested reply and the customer’s sentiment on the agent’s screen.

The challenge wasn’t generating each of these in isolation. It was keeping them in step with a conversation that doesn’t wait for the system. A suggestion that arrives late, belongs to the wrong turn or misses what the customer said is not much help.

We started with the speech pipeline we already had for our voicebot. That got us moving quickly, but we had to rethink quite a bit of it as we turned it into a copilot. Along the way, we also had to work through latency, conversation history, model choices and the cost of running all of this at scale. Production numbers below are from the last 30 days with our internal load-test traffic excluded. Numbers from our test environment are labelled as such.

The latency problem

The agent needs to see the transcript and a useful suggestion while the customer’s response is still fresh.

Our first goal was to get the transcript visible within 2 seconds of the customer finishing, at p95.

Suggestions are harder. They need an LLM call and, sometimes, a knowledge-base search. Our initial target was 4 seconds at p95, with a budget of 700 ms at p99 for the LLM itself. That was a pass mark for not breaking. We have since pushed the goal further. For turns that don’t need a knowledge-base lookup, we want suggestions on screen in under a second at p90.

There is another consideration: how much text we show the agent. Early demos produced suggestions that were too long to read while handling a call. Twenty words turned out to be a reasonable starting point. At a speaking pace of about 160 words a minute, that is roughly 7.5 seconds of speech, one short reply the agent can read at a glance.

Some support workflows need more detail, so we made the limit configurable per assistant, from 10 to 200 words.

We measure latency the way the agent experiences it: from the moment the customer stops speaking to the moment the event arrives on the agent’s WebSocket. Today, a suggestion reaches the agent’s screen in 1.9 seconds at p50 and 2.6 seconds at p90.

Where does the time go?

Here is one actual first turn from production, on a banking assistant running through a contact centre desktop. All timings are measured from the moment the customer stops speaking.

  0.00 s  customer stops speaking
    |
    |  VAD waits for 200 ms of silence
  0.20 s  VAD: speech stopped
    |
    |  streaming STT finalises the transcript
  0.59 s  final transcript ready
    |
    |  fixed end-of-turn wait (0.5 s after VAD stop)
  0.70 s  turn closed
    |
    |  turn sent to the LLM service             ~3 ms
  0.70 s  LLM request starts
    |
    |  suggestion + sentiment generated         0.86 s
  1.56 s  suggestion back in the speech pipeline
    |
    |  gRPC -> backend -> WebSocket             ~1 ms
    |  browser receives and acknowledges        ~40 ms
  1.60 s  suggestion on the agent's screen

Across production turns, the breakdown looks like this:

Stage	p50	p90
Customer stops speaking → final transcript	0.64 s	0.74 s
End-of-turn wait (after VAD stop)	0.50 s	0.53 s
Turn closed → suggestion back from the LLM	1.11 s	1.80 s
Backend hop (gRPC in → WebSocket out)	~1 ms	~1 ms
WebSocket send → browser acknowledgement	43 ms	80 ms
Customer stops speaking → suggestion on screen	1.9 s	2.6 s

Turns that need a knowledge-base answer take a longer path. The model first decides whether to search, then searches, then answers. On production turns of this kind in mid-October, the average was 2.12 s, with 2.43 s at p90:

Step	Average
End-of-turn wait	0.51 s
LLM call 1: decide to search	0.40 s
Knowledge-base search	0.31 s
LLM call 2: write the answer	0.67 s

A suggestion without a search averaged 1.24 s on the same day.

The breakdown pointed to a few things.

Speech recognition is no longer the bottleneck. The transcript is generally ready before the end-of-turn timer expires, so on most turns we are waiting on a timer rather than on the STT engine. This wasn’t always the case. In our initial proof of concept, the first STT engine we tried took about 2 seconds to return English results. The alternative took under 400 ms. Changing the streaming engine moved speech recognition out of the critical path.

We also spend about 700 ms deciding whether the customer has finished speaking. There is 200 ms of silence detection, followed by a fixed 500 ms wait to catch any trailing words.

The LLM accounts for most of the remaining time, and its latency varies quite a bit. On our primary model, time to first token is around 680 ms at p50 but goes up to 1.67 seconds at p90. When we profiled it, each call had a fixed cost of about 0.25 s, plus about 0.04 s for every thousand input tokens. On the fallback model, time to first token is 1.77 s at p50. The fallback exists to protect the experience when the primary is struggling, so it is a problem that it is also the slowest path in the system.

The backend and browser delivery are barely noticeable in comparison. Most of the improvement has to come from end-of-turn detection and the LLM.

What actually improved latency

We spent a focused week trying different approaches on our test environment, measuring each change on its own. Some helped immediately. Others looked promising but did not survive testing.

Change	What happened	Decision
Shorter model output: compact JSON keys, sentiment without its explanation	Output tokens dropped by more than half. Suggestion p90 improved by 430 ms.	Kept
Self-hosted vector search for the knowledge base	Search became roughly a third faster. The knowledge-base prompt shrank from about 8,000 to 3,500 tokens.	Kept
End-of-turn wait 500 ms → 300 ms, with an STT engine that finalises in about 300 ms	Saved around 180 ms on every turn.	Kept
Fewer knowledge-base results (top 5 → top 3)	No meaningful latency improvement. The model was slightly more likely to say it couldn’t answer.	Kept, for the smaller prompt
Smart-turn model for end-of-turn detection	No improvement at p50. End-of-turn p90 went from 0.33 s to 2.12 s.	Reverted
Session affinity for prompt caching	More cache hits, but slower responses under load.	Reverted the same day

Together, the kept changes took a knowledge-base turn from 2.28 s / 2.34 s (p50 / p90) to 1.75 s / 2.04 s on the test environment. A turn without a search went from 1.10 s / 1.22 s to 0.94 s / 1.02 s.

The biggest improvement came from simply asking the model to write less.

LLMs generate output one token at a time. A longer response means more time waiting for the model to finish. We compacted the JSON response, removed the explanation from sentiment output and kept only what the agent actually needs.

It sounds obvious, but this made a bigger difference than most of the infrastructure changes we tried.

The two changes we rolled back taught us more than the ones we kept.

The smart-turn model

The fixed end-of-turn wait is crude. It takes the same amount of time whether the customer has clearly finished a question or has just paused in the middle of a sentence.

A smart-turn model should be able to do better. It listens to the audio and predicts whether the speaker has finished their thought.

We had actually enabled an open-source smart-turn model months earlier. It wasn’t doing anything.

The configuration flag was set, but AI Assist sessions were missing the turn-taking configuration that the model needed. Every turn was still going through the fixed timer. Nothing failed, nothing in the logs told us the model wasn’t running, and our dashboards showed exactly the latency we were already getting.

We have since added instrumentation to record which path closed every turn.

When we finally tested the model properly, against the shorter 300 ms fixed wait, the results were disappointing.

We expected it to reduce the waiting time on complete questions and avoid splitting sentences unnecessarily. Instead, it had a much longer tail than the fixed timer, and it split no fewer sentences.

When the model wasn’t confident, it fell back to a 2-second timeout. This happened on roughly one complete question in nine, and each time the agent would have waited more than two seconds for a suggestion. We switched it off and went back to the fixed timer.

Sentence splitting remained a problem, but it affected suggestion quality more than latency. We are addressing it where it shows up, after generation, by merging the suggestion cards when a question arrives in two pieces, rather than making turn detection more complicated.

Session affinity

Our LLM provider caches prompt prefixes, and about 59% of our input tokens were already being served from cache. Since every turn in a call shares the same system prompt and conversation history, routing all requests from a call to the same model server looked like an easy way to raise that number.

This did increase cache hits.

The problem was that a session pinned to a busy server could no longer move to an idle one. Under load, waiting in a queue cost more time than we saved through caching. On our traffic, a cache miss on a free server was cheaper than a cache hit in a queue.

We reverted the change the same day.

Why we removed the interruption logic

To get AI Assist off the ground, we started with the speech pipeline we already had for our voicebot rather than building another one from scratch. This gave us streaming audio, speech recognition, VAD and much of the turn-management machinery we needed.

It was a good starting point. But it also meant carrying over some assumptions that were specific to how a voicebot works.

A voicebot needs to handle interruptions carefully. When a caller talks over the bot, the bot must stop speaking, cancel its pending response and figure out how much of its previous sentence the caller heard.

Our pipeline had quite a bit of logic for this:

A gate that dropped user audio while the bot was speaking.
Handlers that cancelled in-flight LLM responses when a new turn started.
An estimate of how much of a response had been spoken, from elapsed time and speaking rate.
A counter that ended the call after too many noise-triggered interruptions.

We carried this behaviour into AI Assist.

The assumption was that if a new turn started before the previous suggestion arrived, the old suggestion was no longer useful and should be dropped.

That turned out to be wrong.

On a busy call, a customer often starts speaking again before we have generated a suggestion for what they just said. The pipeline would discard the pending suggestion and send an empty suggestion event.

Further downstream, our backend had a fallback that filled an empty suggestion with the transcript. The agent would sometimes see the customer’s own words in the suggestion box.

There was another issue. Text from unanswered turns kept accumulating, so each new LLM request included more of the previous conversation than it needed.

The underlying mistake was treating a copilot like a voicebot.

A voicebot speaks to the customer. A stale response can be actively harmful because the bot might say something that is no longer relevant.

AI Assist shows text to a human. The agent decides what to use. A suggestion for the previous sentence is often still useful, especially when the customer is continuing the same thought.

We removed the interruption logic from the AI Assist path and replaced it with a much simpler approach. The change deleted about 1,100 lines and added about 130.

The new behaviour is:

A suggestion that arrives after the next turn has started is still delivered, tagged with the turn it belongs to.
At most one unanswered earlier turn is carried into the next request. Text no longer accumulates indefinitely.
The transcript event always contains only the current turn’s text.
Turns without a transcript, such as coughs or line noise, are skipped. There is no reason to call the LLM for them. This happens on about 1% of turns.

The important distinction is that a voicebot has to decide what to say and when to stop saying it. A copilot gives the agent information and lets them decide what to do with it. That difference changes quite a bit of the underlying engineering.

Managing conversation history

The speech pipeline sends one turn at a time to the LLM service. The LLM service maintains the session history and builds the prompt for every request.

The first version simply included everything: every earlier transcript and every knowledge-base result.

There is a good reason for doing this. A customer saying “yes, that one” makes very little sense without the previous conversation.

But the prompt keeps growing as the call progresses.

Each spoken line adds roughly 65 tokens. Over a four-minute call, the prompt grows from about 2,000 to 4,600 tokens, and the LLM call slows from about 0.32 s to 0.65 s. The agent is dealing with the same customer and asking for the same kind of suggestion, but getting it more slowly.

Knowledge-base results make this worse. Some prompts were crossing 8,000 tokens.

In production, the median request carries around 3,200 input tokens, while the p90 is close to 13,000. The actual suggestion is usually only about 75 tokens.

We were sending a lot of history to generate a relatively small response. And a suggestion late in a long call was slower than one at the start, which is the opposite of what an agent dealing with a difficult customer needs.

We are now rolling out a different approach.

The model keeps the last five turns in full and replaces everything older with a rolling summary.

A smaller, cheaper model refreshes this summary every five turns, in the background. The suggestion request does not wait for the summary to be generated.

This keeps the prompt size relatively constant throughout the call.

The trade-off is precision on details from early in the call. A summary may lose an exact figure the customer quoted. It preserves the customer’s issue and commitments made during the conversation, which is what the agent generally needs. The agent has heard the full conversation. The model needs enough context to make the next suggestion useful.

We also made two smaller changes:

Turns where only the agent speaks stay in the history but do not trigger a suggestion.
Filler turns such as “hmm”, “okay” or “one second” return an explicit no-suggestion response. We don’t need to refresh the panel for every bit of speech.

Choosing the models

We evaluated an open-weight model, Gemma 4, which we could run on our own infrastructure, against GPT-4o mini on the jobs AI Assist needs to do: sentiment detection, capturing numbers correctly, producing valid JSON, avoiding hallucinations, following instructions and choosing the right tool. We used 80 to 150 cases per dimension across English, Hindi and Hinglish:

Dimension	GPT-4o mini	Gemma 4
Sentiment accuracy (p95 latency)	84.6% (0.59 s)	91.2% (0.34 s)
Capturing numbers correctly (p95 latency)	86.2% (0.68 s)	96.4% (0.38 s)
Valid JSON output	94.2%	98.8%
Free of hallucination	79.0%	93.8%
Instruction following (judged)	81.4%	85.9%
Choosing the right tool	0.968	0.911

We looked at more than the overall scores. Getting an account number wrong, for example, is more damaging than a slightly less polished response, so the number-capture row carried more weight than instruction following. Latency mattered just as much, because every one of these calls sits inside the budget from the first section. And the cost of running the model at our call volumes mattered, because that cost eventually reaches the customer. A marginal improvement on one task has to justify what it adds to the price of an AI Assist minute.

Gemma performed better on most of these tasks and was faster. GPT-4o mini was better at choosing the right tool. Gemma also tended to produce longer responses, which we addressed later by tightening the output format.

Taking all of this together, Gemma was the better fit as our primary model. We kept GPT-4o mini as the fallback, so we had another option when the primary model was unavailable or struggling. Sizing tests showed two H100 GPUs handling 100 calls a minute at about 6,000 input and 100 output tokens per request, with a p50 of about 275 ms.

Reducing the number of LLM calls

Cost was another reason to revisit the architecture.

Initially, the LLM accounted for close to 60% of the cost of running an AI Assist minute. Each turn was making three separate LLM calls in parallel.

We looked at which calls actually needed to be independent and combined them into one structured request. Sentiment, for example, is now generated along with the suggestion instead of requiring a separate call. That is why the waterfall above shows a single LLM step for suggestion and sentiment.

This reduced both cost and the amount of coordination required between requests.

Speech recognition

Choosing the STT engine required a slightly different evaluation, because raw word error rate wasn’t enough.

In English, our chosen engine had a WER of 0.178 against 0.288 for the next best option. In Hindi, it scored worse on paper, 0.533 against 0.427.

When we listened to the actual audio, many of the apparent Hindi errors came down to issues with the reference transcripts and audio quality. The chosen engine was also better at preserving digits. That matters more to an agent than getting every filler word right. A wrong account number or amount is much more damaging than a missed “um” or “hmm”.

We also looked at what the choice would mean for customers. Speech recognition runs on every minute of every call, so even a small difference in per-minute cost adds up at scale, and at our volume the chosen engine cost less than half its list price. Looking at actual call quality, digit handling and the economics together, it was the better fit for Hindi too.

Masking personal information with an LLM

Call transcripts contain phone numbers, account numbers, card details and addresses. We need to mask sensitive information before a transcript is displayed or stored.

Rules and regular expressions were the obvious starting point. They are fast, cheap and predictable.

But they become difficult to maintain when conversations happen in multiple languages and personal details are spoken rather than typed.

Our calls include English, Hindi, Hinglish and Arabic. A customer might say a phone number digit by digit, use words instead of numbers, or split an account number across a sentence.

A language-aware model handles these cases better than a collection of patterns.

What needs to be masked also varies by customer.

A lender may want loan account numbers masked. A healthcare provider may want conditions and medication details masked.

We allow customers to describe these requirements in plain language, for example:

“Mask the customer’s policy number and date of birth.”

These instructions go directly into the masking prompt. The model identifies matching text and replaces it with asterisks of the same length.

Open-weight models did well here. They handled the language and context variations we needed, and masking runs on the same open-weight model as suggestions by default. This kept the cost manageable without adding a separate commercial API call for every turn.

The important part is how we handle failures.

An LLM can time out or fail, and we cannot allow a masking failure to result in unmasked personal information being exposed.

Our rules are:

During a call, only new customer turns are masked, so the cost grows with new turns and not with call length. The speech pipeline holds the transcript until the masked version is ready. If masking does not return, the transcript is dropped. It is never sent unmasked.
After the call, if masking a stored transcript fails, the raw transcript is encrypted using a key unique to that conversation. A scheduler retries masking every 15 minutes, up to three times.
We alert when the masking failure rate goes above 5%.

We have deliberately chosen to be conservative here. A dropped transcript can be recovered from the recording. Exposed personal information cannot be taken back.

The contract with the agent desktop

AI Assist runs inside multiple agent desktops and CRMs. We wanted the browser integration to be as small as possible.

The browser opens a WebSocket and sends its authentication token as the first message. If the token does not arrive within the expected time, we close the connection.

After that, the browser only receives events.

The first response tells the panel whether the stream is connected and which features are enabled:

{
  "type": "ack",
  "stream_state": "connected",
  "config": {
    "transcript": true,
    "suggestions": true,
    "sentiment": true,
    "summary": false,
    "suggestion": { "enabled": true, "max_words_per_reply": 20 }
  },
  "events": []
}

A transcript event contains the text for one turn:

{
  "events": {
    "event_type": "transcript",
    "turn_number": 3,
    "text": "I want to change my plan",
    "value": [{
      "sequence": 3,
      "transcript_segments": [{
        "speaker": "user",
        "text": "I want to change my plan",
        "start_timestamp": "2026-10-22T10:01:02.120Z",
        "end_timestamp": "2026-10-22T10:01:04.010Z",
        "is_final": true
      }]
    }]
  }
}

The corresponding suggestion event looks like this:

{
  "events": {
    "event_type": "suggestion",
    "turn_number": 3,
    "text": "Sure, which plan would you like to move to?",
    "value": { "text": "Sure, which plan would you like to move to?", "sequence": 3 }
  }
}

Every event carries a turn_number.

This became particularly important after we removed the interruption logic. A suggestion may arrive after the next turn has already started. The turn number lets the panel attach it to the right part of the conversation rather than assuming that the latest event always belongs to the latest transcript.

The browser also acknowledges each event. We use that acknowledgement to measure the time taken for the last leg of delivery.

Stream states and back-pressure

Each live call holds a speech stream, a WebSocket and a share of the available LLM capacity.

LLM capacity is the tightest constraint. In load tests, LLM latency roughly doubled at 10 concurrent requests and rose three- to fourfold at 20. Our own services had problems too.

In the first load test, at just 8 calls a minute, most calls were experiencing more than 10 seconds of input latency. The problem wasn’t the LLM. We had synchronous calls running inside asynchronous code, blocking the event loop for every session handled by that worker.

We fixed this by moving to one process per worker, using pooled HTTP connections to the LLM service so there is no fresh TLS handshake per turn, and rebuilding the affected components on async-native frameworks.

After these changes, the input latency breaches dropped to well under 1% and then to zero. A later 60-minute run at 100 calls a minute pushed 6,000 calls through with 0.02% mid-call WebSocket drops and no latency breaches.

We now enforce concurrent session limits at both the account and platform levels. The limits are checked when a stream is established, never in the middle of a call.

The panel also receives an explicit reason for its current state:

State	Meaning for the agent
`pending`	Stream is being set up
`connected`	Transcript and suggestions are flowing
`throttled`	Capacity limit reached; the next call will try again
`quota_exhausted`	Account’s AI Assist quota has been used up
`connection_timeout`	Stream did not come up in time
`disconnected`	Stream has ended

One important design decision: AI Assist works on a copy of the call audio. Whatever happens to the AI Assist connection, it does not affect the actual conversation between the customer and the agent.

Where we’re focusing next

Getting the suggestion onto the screen is only part of the job. It also has to understand what is being discussed and give the agent something useful to work with. That’s where we’re putting more of our attention now.

LLM latency

Our median latency is getting closer to where we want it, but the tail remains a problem. The model provider’s p50 moves between 0.33 s and 0.72 s through the day, with spikes of 1.3 to 5 s and the occasional 7 s timeout.

The same endpoint is shared with our voicebot. During an incident in October, AI Assist suggestions slowed down at exactly the same time as voicebot responses.

We are looking at two improvements. The first is to combine the two LLM calls on knowledge-base turns into one request, which should save 0.35 to 0.45 s. The second is a hedged retry: when the first request has not returned after about a second, send another and use whichever response arrives first.

End-of-turn detection

The fixed wait after VAD detects silence adds 300 to 500 ms to every turn, and the smart-turn model we tried made the tail worse. We don’t yet have a turn detector that beats a timer on the calls we see. Until we find one, the fixed timer remains the safer choice.

Mono audio

We receive both sides of the call mixed into a single mono stream. One engine we tested labelled every speaker as the same guest, and others give speaker IDs that only mean something within one session. Mono audio also causes an echo loop: when the agent reads a suggestion aloud, it comes back in as speech and can trigger another suggestion. We are testing STT with built-in diarisation, plus a small model that maps speakers to agent and customer. Stereo audio would make most of this go away.

Suggestion quality

Latency is easy to measure. We know exactly when the customer stopped speaking and when the suggestion reached the browser.

Knowing whether the suggestion was actually useful is harder.

We currently use an LLM judge to compare suggestions against the customer’s knowledge base. Across repeated evaluation runs, the share of suggestions judged correct ranged from just over 60% to nearly 100%, which says as much about the judge as about the suggestions.

Agents can give suggestions a thumbs up or down, but that only tells us part of the story.

What we really want to know is whether the agent used the suggestion, ignored it or rephrased it before responding. We don’t capture that yet.

What we’d change if we started again

There are a few things we could have caught earlier.

Reusing the voicebot stack

Starting with the voicebot speech pipeline was the right call. It let us reuse streaming audio and turn management rather than building that foundation again. But we should have reviewed each inherited behaviour against the copilot use case before switching it on. The interruption logic is a good example. A one-page list of what changes when nothing is spoken back would have saved us a round of bugs.

Checking whether something is actually running

The smart-turn model was configured but never executed. We were looking at latency dashboards that told us how long turn detection took, without telling us which turn detection mechanism had actually run. A per-turn field showing which path closed the turn would have caught that on day one.

Tracking prompt size from the first call

Prompt growth was a gradual problem. Nothing failed, but suggestions became slower as conversations progressed. The same was true of verbose model output, which cost us 0.4 s at p90 before we looked. Tracking input and output tokens by turn would have helped us catch both much earlier.

We started with the voicebot stack to get moving quickly. A lot of the work since then has been about changing those foundations to suit a copilot rather than a bot that speaks back. The next challenge is making the suggestions more relevant to the conversation, so agents get help they can actually use.