AI system design interview: design a voice agent for appointment booking
Connect telephony to a streaming pipeline: turn detection, streaming speech-to-text, a small fast LLM with calendar tools (find slots, book, move, cancel), and streaming text-to-speech, with a latency budget per stage. Read back the date and time before confirming, require explicit confirmation for cancellations, and transfer to staff for anything medical or when the caller asks. Test with simulated callers and replayed real calls for dates, interruptions and accents.
Question
Design a phone agent that books, moves and cancels clinic appointments.
Clarify
- Call volume and peak hours.
- Languages and accents.
- Calendar system and its API.
- What the agent must never do: give medical advice, book without confirmation, cancel the wrong appointment.
- Recording consent rules.
Latency budget
A natural reply starts within about a second of the caller finishing.
| Stage | Budget |
|---|---|
| Turn detection (caller finished speaking) | 200 to 300 ms |
| Speech-to-text final result | 100 to 300 ms |
| LLM first token | 300 to 500 ms |
| Text-to-speech first audio | 100 to 200 ms |
These are design targets to measure against, not guarantees, and tool calls to the calendar add to them, so a short filler ("let me check that") can cover a slow lookup.
Architecture
| Component | Role |
|---|---|
| Telephony | Phone number, call handling, transfer to staff |
| Turn detection | Decide when the caller has finished, handle interruptions |
| Streaming STT | Transcribe as the caller speaks |
| LLM with tools | Find slots, book, move, cancel, identify the patient |
| Streaming TTS | Speak as text is generated |
| Confirmation | Read back date, time and clinician before any change |
| Handoff | Transfer with a summary for medical questions or on request |
| Logging | Transcripts and tool calls, with consent |
Deep dive: dates
"Next Tuesday", "the 3rd" and "after Diwali" are where voice agents fail. Resolve dates in code from the current date and the clinic's calendar, read the resolved date back, and only call the booking tool after the caller confirms.
Evals
Simulated callers with scripted scenarios (moving an appointment, interrupting, changing their mind), replayed real calls where consent allows, and metrics: booking accuracy, confirmation compliance, transfer rate and call duration.
What interviewers listen for
A latency budget per stage, confirmation before changes, and handoff for anything outside scope.
From the floor
In the Echo Negotiator build in AI Engineer HQ, we built a voice-to-voice AI for conversations like negotiations and sales calls, with real-time full duplex communication, where both sides can speak at the same time, so that it feels closer to talking to a real person, and handling interruptions is what makes that hard.