Online evaluation and A/B tests for LLM features
Use offline evals to decide whether a change is safe to ship, and online experiments to decide whether it's better for users. Pick one primary business metric (task completed, resolved without escalation, conversion), a few guardrail metrics (cost, latency, complaint rate), decide the sample size and duration before you start, and randomise by user, not by request. Expect noisier results than with normal features, because LLM outputs vary, and use shadow mode when traffic is too small for a fair test.
Offline and online evaluation
Offline evaluation happens before release, on a fixed test set: does the new version pass the checks? Online evaluation happens in production, with real users: does the new version change what users do?
You need both. Offline evals answer "is it wrong?". Online tests answer "does it matter?".
A new prompt can win on the offline eval and lose in production, because the eval measured correctness and users cared about speed, or because the new answers are correct and twice as long.
What to measure online
| Kind | Examples |
|---|---|
| Primary metric (one) | Task completion, resolved without human escalation, conversion, retention |
| Quality signals | Thumbs down rate, regeneration rate, edits to drafts, handoffs |
| Guardrail metrics | Cost per session, p95 latency, complaint rate, error rate |
| Background evaluators | Pass rate of validated judges on sampled traffic |
Choose one primary metric before the experiment starts. With five primary metrics, one of them will move by chance and look like a win.
Designing the experiment
- Randomise by user, not by request, so one person doesn't see both versions in the middle of a task.
- Decide the sample size and duration before starting, based on the smallest effect you care about and how much your metric varies.
- Run for full weeks, because behaviour changes between weekdays and weekends.
- Watch guardrail metrics daily, and stop early if cost or complaints spike.
- Don't stop the moment the result looks good. Peeking and stopping early inflates false wins.
import math
def users_per_arm(baseline: float, lift: float, z_alpha: float = 1.96, z_beta: float = 0.84) -> int:
p1, p2 = baseline, baseline + lift
p_bar = (p1 + p2) / 2
num = (z_alpha * math.sqrt(2 * p_bar * (1 - p_bar)) + z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
return math.ceil(num / lift**2)
for lift in (0.05, 0.02, 0.01):
print(f"detect +{lift * 100:.0f} points on a 60% resolution rate: {users_per_arm(0.60, lift):,} users per arm")
Small improvements need a lot of users to detect. If you don't have that traffic, use the options below.
When traffic is small
- Shadow mode: run the new version silently on real requests, don't show its answers, and compare its outputs offline with your judges and your expert.
- Interleaving: for search and ranking, mix results from both versions in one list and see which ones users pick.
- Staged rollout: release to 5% of users, watch the guardrail metrics, then expand. This is about safety more than measurement. See rollouts and model upgrades.
Write the plan first
An experiment plan has:
- Hypothesis: "Showing sources under each answer will reduce escalations."
- Primary metric and the minimum effect worth shipping.
- Guardrail metrics and their limits.
- Sample size and duration.
- Stop conditions.
- Who decides, and what happens for each outcome.
From the floor
In the ML system design session, evaluation had two parts: offline evaluation before deployment, and online evaluation after it, measured by what users actually do. For a video recommendation system, the online measure I used was whether users completely watched the recommended videos, and for harmful content detection, whether the content flagged was actually harmful, because the business result is what tells you the model is good, not only the technical metric.