AI system design interview: how to prepare, with 25 worked questions
An AI system design interview asks you to design an LLM-based system out loud, usually in 45 to 60 minutes. Clarify requirements and constraints, define what correct means and how you'd measure it, decide whether the problem needs an LLM at all, sketch the architecture in zones, go deep on two or three components, and finish with failure modes, cost and what changes at 10 to 100 times the scale. Interviewers listen for trade-offs and evaluation, not for tool names.
How it differs from classic system design
Classic system design focuses on storage, throughput and consistency. AI system design keeps all of that and adds problems classic systems don't have: outputs that vary between runs, answers that sound right and are wrong, cost per request that depends on prompt length, latency measured in seconds, and actions taken by a model on someone's behalf. So interviewers want to hear how you'd evaluate the system, where exact values come from, and what the model is allowed to do.
A seven-step answer structure
1 - Clarify: users, scale, latency, budget, privacy, languages, and what the system must never do. 2 - Define success: the business metric, and the eval you'd use to know the system meets it. 3 - Decide whether it needs an LLM, and where: rules, search or a script may solve parts of it. 4 - Sketch the architecture in zones: gateway and cache, input guardrails, orchestration, retrieval, inference, output validation, observability. 5 - Go deep on what the interviewer cares about: retrieval, the agent loop, routing, memory. 6 - Walk through failure modes and how you'd detect and limit each one. 7 - Cost, scaling, and what you'd build first.
Spend the first 5 to 8 minutes on steps 1 and 2. Drawing before you know the constraints is the most common mistake.
Estimate with numbers
Do quick back-of-envelope math out loud: requests per day, tokens per request, cost per request, peak requests per second, GPU memory if self-hosted. For example, 1 million requests a day at 3,000 input and 300 output tokens is 3 billion input and 300 million output tokens a day, so you can multiply by the per-token prices of the models you're considering. See capacity and cost modelling.
What interviewers listen for
- You ask about requirements before drawing.
- You say how you'd evaluate it, and you separate retrieval quality from answer quality.
- You know where exact values come from: databases and tools, not the model.
- You add approval where actions are risky or irreversible.
- You estimate cost and latency with numbers.
- You name a limitation of your own design.
Common mistakes
- Starting with a framework name instead of the problem.
- Adding an agent where a fixed workflow would do.
- No evaluation plan, or "we'd use an LLM judge" with no check on the judge.
- Treating the prompt as a security boundary.
- Ignoring cost until the interviewer asks.
How to practise
Pick a question below, set a 45-minute timer, and answer out loud while drawing. Then compare your answer with the worked one and note what you missed. Better still, build a small version of one: an answer you can back with "when we tested this, here's what broke" is much stronger than one from memory.
Questions
From the floor
Several of these questions came from builders in live sessions: in the inference pipeline lecture, a builder asked how to manage session memory for 10,000 users who come back a week later, including the context bleeding they'd seen in production, and I said I didn't have a complete answer ready in the session, which is why question 19 is the fuller answer.