AI system design: a framework for designing LLM systems
Design an AI system in seven steps: state the business objective, frame the AI task (inputs, outputs, what correct means), prepare the data, choose and develop the model approach, define offline and online evaluation, design the deployment, and plan monitoring. Then draw the architecture as zones (gateway and cache, guardrails, orchestration, retrieval, inference, validation and observability) and decide what each zone needs for your requirements. Keep it as simple as the problem allows: start from the user's request, and add a component only when a requirement or a failure asks for it.
Why start from the problem, not the model
Most AI architecture diagrams start with the model and work outward. The result is a system that answers questions nobody needed answered, at a cost nobody planned for.
Starting from the problem changes the questions you ask: what metric moves if this works, what does the user send, what must come back, how fast, how often, and what happens when it's wrong. Sometimes the answer to those questions is that a script, a search box or an existing product solves it, and that is good engineering too.
System design basics still apply
AI systems are software systems first. The qualities every system design starts with still hold:
- Scalability: the system handles more users and more data without falling over.
- Reliability: it keeps working when parts fail.
- Performance: it responds fast enough for its users.
- Maintainability: people can understand it, change it and fix it.
And the common components are still there: load balancers spreading requests across servers, caches, queues, databases, API gateways. AI adds its own components on top: vector stores, inference servers, routers, guardrails, evaluators. For AI systems, cost per request and quality on an eval set join the list of qualities you design for.
Seven steps
1. Business objective
What metric moves if this works? Fewer support tickets reaching people, faster contract review, more bookings, fewer errors in invoices. If you can't name the metric, stop here, because you won't be able to tell whether the system works.
2. Frame the AI task
Inputs, outputs, constraints, and what a correct output is. "Given a customer's message and their order history, either answer from policy with a citation, take one of three actions, or hand off to a person." Decide here whether this needs an LLM at all.
3. Data
What data the system reads, where it lives, who can access it, how fresh it must be, how it gets in, and what personal or regulated data it contains. See data architecture.
4. Model approach
Prompting, retrieval, tools, fine-tuning, or a mix, and which model sizes for which steps. See model adaptation and model routing.
5. Evaluation
Offline evals before launch, online metrics after. Accuracy alone can mislead: a model can be 99% accurate and still miss the 1% of cases that matter most, so define what failure means for this product. See evals and reliability.
6. Deployment
Serving, scaling, latency budgets, cost limits, security, and where it runs (cloud, on-premises, device). See LLMOps.
7. Monitoring
Tracing, quality sampling, drift, cost, and the loop back to step 3 or 4 when something changes.
One-page design template
Objective: metric and target
Task: input -> output, definition of correct
Non-functional: latency p95, cost per request, availability, privacy
Data: sources, freshness, access rules
Approach: components and models per step
Evaluation: offline set, online metric
Deployment: where it runs, scaling, limits
Monitoring: what is traced, what alerts
Risks: top 3 failure modes and mitigations
A design is good when someone else can build it from the page, and when its first month in production surfaces no failure mode you didn't list.
From the zones to the details
Once the seven steps are clear, draw the architecture. The LLM inference pipeline divides a production LLM system into six zones, and the patterns catalogue gives you building blocks for each one.
What this section covers
| Page | The question it answers |
|---|---|
| LLM inference pipeline | What happens between pressing enter and getting an answer? |
| LLM systems architecture chapters | Where is each topic of the lecture series covered? |
| Patterns catalogue | Which reusable patterns solve which problems? |
| Reference architectures | How are common AI products built? Support assistant, copilot, knowledge engine, document processing, agent platform, voice assistant |
| Non-functional requirements | How do I write testable targets? |
| Capacity and cost modelling | How many GPUs, how much money? |
| Multi-tenant platforms | How do I serve many customers safely? |
| Data architecture | Which data stores does an AI system need? |
From the floor
In the first AI system design session, I said that after building many architectures, I've found there's no single right way to draw one, and the right way is to keep it simple: start from the problem, and even a beginner who thinks it through gets close to the solution, while experience adds the details you'd otherwise miss. We started with three foundational components, a load balancer, a vector store and an inference server, and built a small simulation of each in Python in the next session.
In the ML system design session, I taught the same seven steps for any ML system: requirements, framing the problem as an ML task, data preparation, model development, evaluation, deployment and monitoring, with evaluation split into offline and online parts.