Reference architecture: voice assistant
A voice assistant connects a phone line or browser audio to a streaming pipeline: voice activity and turn detection, streaming speech-to-text, an LLM with tools and a short prompt, streaming text-to-speech, and barge-in handling so callers can interrupt, with a transfer to a person, and recordings and transcripts for evaluation. Every stage streams, because the whole budget from the caller stopping to the assistant starting to speak is well under a second. How voice agents work is covered in voice agents, and this page is the system around them.
Architecture
caller (phone or browser)
│ audio stream (telephony provider or WebRTC, WebSockets to your server)
▼
voice activity + turn detection ──> streaming speech-to-text
│
▼
LLM (small, fast) + tools + pre-fetched customer data
│
▼
streaming text-to-speech (first sentence first)
│
caller <─────────────── audio ───────────────┘
│
├── barge-in: caller speaks → stop audio, cancel generation, listen
├── handoff: warm transfer to a person with a summary
└── recording, transcripts, traces → evals and compliance
Components
| Component | Role |
|---|---|
| Transport | Telephony provider or WebRTC, streamed over WebSockets |
| Turn detection | Decides when the caller has finished speaking |
| Speech to text | Streaming transcription |
| LLM and tools | A small, fast model, with customer data fetched before the call is answered |
| Text to speech | Streaming voice, starting on the first sentence |
| Barge-in | Stops speaking and listens when the caller interrupts |
| Handoff | Warm transfer to a person with a summary |
| Recording and transcripts | Compliance and evals |
Key decisions
- Cascaded pipeline (speech-to-text, LLM, text-to-speech) or a native speech-to-speech model. The pipeline gives more control and logging, speech-to-speech can be faster and more natural.
- The latency budget per stage, and which model fits it.
- What the assistant says while a slow tool runs.
- Consent, recording and disclosure rules per region, including telling callers they're speaking with an AI.
Failure modes to test
- Cutting the caller off mid-sentence, or waiting too long after they finish.
- Not stopping when the caller interrupts.
- Misheard numbers, names and dates. Confirm them out loud.
- Background noise and poor phone lines.
- Long silences while tools run.
Metrics
- Time to first audio after the caller stops speaking.
- Task completion per call.
- Transfer rate to people.
- Interruption handling success.
- Transcription accuracy on numbers and names.
From the floor
Echo Negotiator, the third artifact in AI Engineer HQ, was a real-time, full duplex voice-to-voice server: FastAPI with WebSockets for streaming, Silero for voice activity detection so the AI stops when the user interrupts, faster-whisper for speech-to-text and a local LLM through Ollama, designed to run fully offline with responses under 500 milliseconds as the goal. The problem statement was that voice bots feel unnatural, respond slowly and can't handle interruptions, which rules them out of fast, high-stakes conversations like negotiations, sales calls, coaching and interviews.