AI system design interview: reduce latency for a chat assistant
Measure latency per stage first, and separate time to first token (what users feel) from total time. Then stream tokens, cache frequent answers, run retrieval and independent tool calls in parallel, use small models for routing and guardrails, shorten the prompt, move slow checks off the critical path, and serve from a region near users. Re-run evals after each change, because smaller models and less context can cost quality.
Question
Users say the assistant feels slow: p95 is 9 seconds. Bring it down.
Clarify
- Is it slow to start answering, or slow to finish?
- Is the 9 seconds at p95 caused by a few long requests (agent loops, long documents) or by every request?
- Which model and hosting: an API or self-hosted?
First, measure
Trace each request and time every stage: gateway, cache lookup, input guardrails, query rewriting, retrieval, reranking, model time to first token, generation, output checks. A typical finding is that sequential steps before the model (rewriting, retrieval, reranking, guardrails) add several seconds before the first token appears.
Levers
| Lever | Helps | Trade-off |
|---|---|---|
| Stream tokens to the UI | Perceived latency | Output checks must work on streams or run after |
| Cache frequent answers | Hits return in milliseconds | Wrong hits |
| Parallel retrieval and independent tools | Total time | More concurrent load |
| Small models for routing, rewriting and guardrails | Time before the main model | Lower accuracy on those steps |
| Shorter prompt and context | Time to first token, cost | Missing context |
| Move slow checks to async evaluators | Critical path | Problems found after the user sees them |
| Region near users | Network time | Data residency rules |
| Speculative decoding, batching tuning (self-hosted) | Generation speed | Engineering effort |
Time to first token vs total time
Time to first token depends mostly on what happens before generation and on the length of the prompt. Total time depends on output length and tokens per second. So cutting prompt length and pre-model steps fixes the first, and capping output length or using a faster model fixes the second.
Trade-offs
Smaller models and less context can lower quality, so re-run evals after each change and keep the ones that hold quality.
What interviewers listen for
Time to first token vs total time, per-stage numbers, and quality checks after each change.
From the floor
In the inference pipeline lecture, I walked through the request path zone by zone, the gateway and semantic cache, then the input guardrails like PII masking and injection checks, then the model, and every one of those zones is a place where latency adds up before a user sees the first token, which is why you time each zone separately.