AI system design interview: design a customer support agent
Route each message by intent, answer policy questions from versioned documents with citations, get order data only from tools called with the customer's own scoped access, and hand off to a person with a summary when confidence is low, the customer is upset, or an action needs approval. Measure resolution and correct handoffs, not just answer quality.
Question
Design an AI agent that handles customer support for an e-commerce company with 2 million orders a month.
Clarify
- Channels: chat only, or email too?
- Languages.
- Actions: order status yes, refunds maybe, cancellations with approval?
- Latency target for chat, for example first token under 1 second.
- What must never happen: promising refunds outside policy, exposing another customer's data, making up delivery dates.
Estimate
If 5% of orders lead to a support conversation, that's 100,000 conversations a month, about 3,300 a day. At 6 turns and about 4,000 tokens per turn including context, that's around 80 million tokens a day, so routing simple questions to a small model and caching policy answers matter for cost.
Architecture
| Zone | Component |
|---|---|
| Entry | Authenticated chat, session tied to the customer |
| Gateway | Rate limits, exact and semantic cache for policy FAQs |
| Input guardrails | PII masking, injection checks |
| Orchestration | Intent router: FAQ, order question, refund, complaint, handoff |
| Retrieval | Policy RAG over versioned documents, with citations |
| Tools | Order lookup, tracking, refund request, all scoped to the customer's token |
| Output checks | Schema validation, policy claims checked against retrieved text |
| Handoff | Ticket with conversation summary and the agent's attempted actions |
| Observability | Tracing per turn, feedback linked to traces |
Deep dive: router and handoff
The router decides both cost and safety. FAQs can go to a small model with cached answers, order questions need tools, and complaints or angry messages go to a person early. Refunds above a threshold, cancellations and anything outside policy need a person's approval. Handoff includes a summary, so the customer doesn't repeat themselves.
Failure modes
| Failure | Mitigation |
|---|---|
| Policy hallucination | Answer only from retrieved policy, cite it, abstain otherwise |
| Wrong order data | Values from tools only, never from the model's memory |
| Another customer's data | Tools called with the customer's token, not a service account |
| Instruction hijacking over several turns | Multi-turn tests, scope rules in the orchestrator |
| No handoff when needed | Explicit triggers: low confidence, negative sentiment, repeated failure |
Evals and metrics
- Policy accuracy on sampled answers, judged against the policy text.
- Correct handoff rate: handed off when it should, kept when it could.
- Resolution without escalation, and repeat contact within 7 days.
- Cost per resolved conversation.
What interviewers listen for
That the agent's actions are limited and approved, that policy answers are grounded, and that you measure resolution, not just answer quality.
From the floor
In the customer support agent project in AI Engineer HQ, I used Stripe's support pages as the example: billing, invoices, payments, payouts, privacy and refunds are all topics customers ask about, and we built retrieval over that kind of policy data so the agent answers from the company's documents instead of the model's memory.