When a caller stops talking, a human agent responds in about 200–500 milliseconds. Cross one second of silence and the caller says "hello?" — cross two and they hang up. Every voice AI agent we build at Aivetech starts from this constraint, because no amount of conversational intelligence survives an awkward pause.
Where the time actually goes
A naive pipeline — record the caller, transcribe, send the transcript to an LLM, synthesize the reply, play it — stacks each stage end to end and easily lands at three to five seconds. The production version overlaps everything:
- Streaming speech-to-text. Transcription runs while the caller is still speaking, so when they stop, the transcript is already 95% complete. Endpoint detection (deciding the caller has finished) is the real cost here — too aggressive and you interrupt them, too lax and you add 400ms of dead air.
- Incremental LLM inference. We stream tokens and begin synthesis on the first complete clause, not the full reply. Time-to-first-token matters more than total generation time; model choice is dominated by this number for voice workloads.
- Streaming text-to-speech. The first audio chunk should leave the server before the LLM finishes the sentence. Chunk-level synthesis with a small lookahead keeps prosody natural without waiting for a paragraph.
- Telephony overhead. The part nobody budgets: codec transcoding, jitter buffers, and geographic routing quietly cost 100–300ms. Hosting inference close to the telephony edge matters as much as model choice.
Barge-in: the feature that separates demos from products
Real callers interrupt. When they do, the agent must stop talking within roughly 200ms, discard its queued audio, capture what the caller said, and reconcile it against the half-delivered reply. This means the pipeline needs cancellation tokens at every stage — a design decision that has to exist from day one, because retrofitting interruption into a fire-and-forget pipeline is a rewrite.
What we measure
Every agent we ship reports three numbers per call: time-to-first-word after caller silence, interruption recovery time, and task completion rate. Transcript accuracy matters, but callers forgive a misheard word far faster than a three-second pause. If you're evaluating a voice vendor, ask for these numbers before you ask for a demo — the demo is always on a good network.
We go deeper on architecture and integration on our voice AI agents page, or you can book a call and hear a live agent respond within its budget.