The LLM inference pipeline: a production architecture in six zones
A production LLM system does a lot before and after the model runs. I divide it into six zones: (1) API gateway, semantic cache and router, (2) input security and guardrails, (3) agent orchestration with tools and memory, (4) retrieval from external data, (5) model inference, and (6) output validation, safety filters and observability. Inference is the most expensive step, so the zones before it exist largely to avoid it or make it smaller.
Why "the user asks, the model answers" isn't enough
People picture an LLM system as one step: the user asks, the model answers. That picture can't explain why some answers come back instantly and others take seconds, why personal data is masked, where an agent fits, or why the same question costs less the second time.
The main design idea behind the six zones: put as many checks as you can before the expensive step. Every query a cache answers, a router sends to a small model, or a guardrail blocks is a query that never loads billions of parameters on a GPU.
One query through the six zones
The example query: "What is the delivery status of order TS9120, and what is your policy if it arrives damaged?" It has two parts: a live lookup (the order) and a policy question (the damage policy).
user ──> [1 gateway · cache · router] ──hit──> answer
│ miss, complex
▼
[2 PII masking · injection checks]
▼
[3 orchestrator: split into sub-tasks]
│ │
▼ ▼
tool: order database [4 retrieval: hybrid search, rerank, compress]
└──────────┬──────────────┘
▼
[5 inference: prefill, decode]
▼
[6 schema check · safety filter · unmask · trace]
▼
cache updated, answer returned
Zone 1: gateway, cache and router
The request reaches an API gateway (LiteLLM or Kong are examples), the query is embedded, and a semantic cache, Redis is a common choice, is checked. A cache hit returns the stored answer immediately. On a miss, a router judges how complex the query is: a simple one ("what is the capital of India?") goes to a small model that can run cheaply, and a complex one continues. Our example needs a database lookup and a policy lookup, so it continues. See semantic caching and routing.
Zone 2: input security and guardrails
Personal data (emails, card numbers, ID numbers) is masked before anything reaches a model, with a tool like Microsoft Presidio, and a reverse map is kept so the data can be restored in the final answer. Personal data adds nothing to the model's reasoning, and sending it to a model provider creates a privacy risk.
Then the query is checked for prompt injection, jailbreak attempts and policy violations. NVIDIA NeMo Guardrails is one option. Prompt injection is similar to SQL injection: instead of a query hidden in a form field, it's an instruction hidden in text, like a comment that says "forget all your previous instructions". See prompt injection.
Zone 3: agent orchestration
Because the query needs actions, an orchestrator (LangGraph is an example) splits it into two sub-tasks: fetch order TS9120 from the live database, and retrieve the damage policy. Tools come from a secure tool registry and run in an isolated sandbox. Agent state and memory are stored (Mem0 is one option), so every step is recorded. If a query needs no actions, this zone is skipped. See the tool registry pattern.
Zone 4: retrieval
An offline ingestion pipeline has already parsed, chunked and embedded the policy documents into a vector database, and can add a graph of entities and relationships. Online, hybrid search (BM25 keyword matching plus dense vectors) finds candidates, a cross-encoder reranker keeps the most relevant ones, and a context compressor (Microsoft's LLMLingua is an example) trims them before generation, to cut tokens without losing the meaning.
The order status doesn't come from retrieval. It comes from the database through a tool, as an exact value, because the order number and its status must never be generated. See the exact values pattern.
Zone 5: inference
The prompt, the order data and the policy passages go to the model, served by an engine like vLLM or NVIDIA TensorRT-LLM. Prefill processes the whole prompt in parallel and fills the KV cache, managed with PagedAttention, and decode generates the answer one token at a time. Quantized weights reduce the memory each step needs. See inference optimization.
Zone 6: output validation and observability
The output is checked against a schema (Pydantic), so an application that expects JSON gets valid JSON, and if it fails, it goes back to the model. Then it's filtered for leaked system prompts, internal data and toxic content (Llama Guard is one option). Masked personal data is restored, the cache is updated if the answer is cacheable, everything is traced and sent to continuous evaluation (Langfuse or LangSmith), and the answer returns through the gateway.
Using the zones in your own design
Draw your system in these six zones. For each zone, write which component you use, or "not needed" with a reason. A blank zone with no reason is a gap.
Not every system needs every zone. An internal summarisation tool may skip the cache and the agent zone. A public chatbot can't skip zone 2.
What to measure per zone
| Zone | Metric |
|---|---|
| 1 | Cache hit rate, router distribution, routing errors |
| 2 | Guardrail trigger rate and false positives |
| 3 | Tool success rate, steps per task |
| 4 | Retrieval recall, context tokens after compression |
| 5 | Time to first token, tokens per second, cost |
| 6 | Validation failure rate, safety filter triggers |
From the floor
This architecture was the first session of AI System Design Engineering, built from what I've seen in real production systems, and we walked the example query through every zone. Builders asked questions you should be ready for too: is this semantic or prompt caching (semantic caching on the user's question), doesn't an intent router add latency (yes, so you test whether it pays off), what is LLMLingua doing (compressing the retrieved context before it reaches the model, to cut tokens), and how do you keep session memory for 10,000 returning users, where I said it starts at the cache and a long-term memory store and that I didn't have a complete answer ready in that session.