AI system design patterns catalogue
A design pattern is a reusable solution to a problem that keeps coming back. For LLM systems, the most useful ones are the semantic cache, the model router, the guardrail sandwich, the hybrid fleet, the tool registry and sandbox, the exact values pattern, the fallback chain and the async job queue. Use them as building blocks in your designs, and name them in design reviews and interviews, because a named pattern carries its trade-offs with it.
Why patterns help
Every new AI system feels unique, and most of its problems aren't. Paying twice for the same answer, sending easy questions to an expensive model, a model rewriting a number from the database, a provider outage at peak hours, a task too long for an HTTP request: these happen in almost every system.
A pattern gives you a tested answer and a shared name, so a team can say "put a guardrail sandwich around this call" and everyone knows what that means.
Catalogue
| Pattern | Problem it solves | Zone | Page |
|---|---|---|---|
| Semantic cache | Paying repeatedly for the same answer | 1 | Semantic caching |
| Model router | Using an expensive model for easy requests | 1 | Model routing |
| Guardrail sandwich | Unsafe input in, unsafe output out | 2 and 6 | Guardrail sandwich |
| Hybrid fleet | One model can't satisfy cost, privacy and quality at once | 1 and 5 | Hybrid fleet |
| Tool registry and sandbox | Agents calling arbitrary code and APIs | 3 | Tool registry |
| Exact values | The model rewriting numbers and IDs | 3 and 4 | Exact values |
| Fallback chain | Provider outages and bad outputs | All | Fallback chain |
| Async job queue | Long tasks blocking requests | All | Async job queue |
| Human approval gate | Irreversible actions by agents | 3 | Human in the loop |
| Hybrid retrieval | Keyword and meaning both matter | 4 | Hybrid search |
| Workflow patterns | Predictable multi-step tasks | 3 | Workflows vs agents |
The zone numbers refer to the six zones of the LLM inference pipeline.
How to choose
Start from the failure, not the pattern. For each risk in your design, ask which pattern addresses it:
- "Most of our questions repeat." → Semantic cache.
- "Most of our questions are simple." → Model router.
- "We handle personal data, or the public can type anything." → Guardrail sandwich.
- "Some data can't leave our servers, and we still want frontier quality for the rest." → Hybrid fleet.
- "The agent runs code or changes systems." → Tool registry and sandbox, plus a human approval gate.
- "Answers contain numbers that must match our database." → Exact values.
- "We can't afford downtime when a provider fails." → Fallback chain.
- "Some tasks take minutes." → Async job queue.
Each pattern adds components to build, test and maintain. Add one when a requirement or a failure asks for it, not because it appears in a reference diagram.
From the floor
Most of these patterns appeared as parts of the production pipeline in the first AI System Design Engineering session: the cache and router in the first zone, the guardrails before and after the model, the tool registry with a sandbox in the agent zone, and the database lookup for exact values. This page names them as patterns so you can reuse them in your own designs.