Latency budgets and streaming UX for LLM apps
For users, latency is the wait between pressing enter and the first word appearing. Measure time to first token (TTFT) and total time separately, break the request into stages and give each one a budget, stream the output so people can start reading, and show progress for steps that take longer, like retrieval or tool calls. Users forgive a long answer that starts quickly far more than a short one that starts late.
What latency means here
You know the wait after you send a message and the three dots appear. In an LLM app, that gap is the time it takes from the moment you hit enter to the moment the first word of the answer appears.
Two numbers describe it:
- Time to first token (TTFT): how long until the first token appears. This is what users feel as "slow" or "fast".
- Total time: how long until the answer is complete. This matters for long answers, agents and anything another system waits on.
A third number matters for streaming: tokens per second, or how fast the text appears once it starts. People read at a few words per second, so beyond a certain speed, faster streaming doesn't feel different.
Where the time goes
A RAG answer takes 6 seconds. Without per-stage timing, you'd blame the model. With it:
| Stage | Time (seconds) |
|---|---|
| Gateway and auth | 0.1 |
| Cache check | 0.05 |
| Input guardrails | 0.3 |
| Query embedding | 0.1 |
| Retrieval and reranking | 1.2 |
| Prompt assembly | 0.05 |
| Model: time to first token | 0.9 |
| Model: rest of the answer | 3.3 |
| Total | 6.0 |
Now you can see that retrieval and reranking take longer than the model's first token, and that streaming would let the user start reading at about 2.7 seconds instead of 6.
Set a budget per stage
Decide the target first (for example, first token under 2 seconds at p95), then give each stage a share of it. When a stage goes over its budget, you know which team or component to look at.
Measure p50 and p95, not averages. An average of 2 seconds can hide 10% of users waiting 9 seconds.
import time
from contextlib import contextmanager
timings: dict[str, float] = {}
@contextmanager
def stage(name: str):
start = time.perf_counter()
yield
timings[name] = time.perf_counter() - start
with stage("retrieval"):
time.sleep(0.12)
with stage("first_token"):
time.sleep(0.25)
with stage("rest_of_answer"):
time.sleep(0.40)
budget = {"retrieval": 0.10, "first_token": 0.30, "rest_of_answer": 1.00}
for name, seconds in timings.items():
flag = "OVER" if seconds > budget[name] else "ok"
print(f"{name:<15}{seconds * 1000:6.0f} ms budget {budget[name] * 1000:5.0f} ms {flag}")
In production, record these as spans in your traces, so you can chart them per route and per release.
Ways to reduce latency
| Lever | Effect |
|---|---|
| Streaming | Lower perceived latency, the user reads while the model writes |
| Semantic or prompt caching | Skip the model entirely, or shorten prefill |
| Routing to smaller models | Faster answers for simple requests |
| Parallel retrieval and tool calls | Removes waiting between independent steps |
| Shorter prompts | Less prefill, faster first token |
| Shorter outputs (limits, concise instructions) | Less decode time |
| Fewer sequential LLM calls | Each extra call adds a full round trip |
| Region close to users | Less network time |
| Async code | Serve many requests without blocking. See async Python |
Streaming
With streaming, the server sends tokens as they're generated, usually with server-sent events (SSE) or WebSockets. Most provider SDKs support it with one setting.
Things to handle when you stream:
- Output guardrails: you can't validate a full answer before showing it. Check chunks as they arrive, or hold back the parts that need checking (like the final JSON) until they're complete.
- Errors halfway through: decide what the user sees if the stream breaks.
- Structured output: stream the text the user reads, and send structured data in a final event.
Design the UI around waiting
- Show something immediately: a typing indicator, or the user's message echoed.
- Show progress for slow steps: "Searching your documents", "Checking your order".
- Stream text as it arrives.
- Show sources and actions when they're ready, not after the whole answer.
- For long tasks, don't make the user wait in the chat. Run them as background jobs and notify. See the async job queue pattern.
Accuracy or latency
Not every product needs the fastest answer. Decide what matters more for the use case: a fraud check or a voice agent needs speed, and a legal research answer can take longer if it's correct. That decision sets which model, how many retrieval steps and how many checks fit in the budget.
From the floor
When I explain latency, I use the wait we all know: you send a message to a friend and the three dots appear. In an AI app, latency is the time from the moment you hit enter to the moment the first word of the answer appears, which is the AI gathering its thoughts, processing your request and working out what to say first.
In the ML system design session, one of the questions I asked for every system was whether accuracy or latency has priority, because a complex model may be more accurate and need much more compute and time, and the answer depends on what the user needs from that product.