How to pick which LLM traces to read
You'll never read every trace, so you need a way to pick the ones that teach you the most. Combine five samplers: random, clustered, outliers, classifier-flagged and user feedback, and write the mix down, for example 40% random, 30% clustered and 30% flagged. Lean on random and clustered sampling early, when you don't know your failure modes yet, and always keep a random slice, because it's the only part of the batch that can surprise you.
Five ways to sample
Random sampling
Every trace has an equal chance. It's honest, it shows you what typical traffic looks like, and a small random batch misses rare failures.
Clustering
Group similar traces together, usually by embedding the user's request, and pick a few from each group. You see the shape of your traffic instead of 50 copies of the most common question.
Outlier sampling
Sort traces by extreme values: highest latency, most tool calls, most retries, longest conversations, highest cost. Agents often loop or give up in those traces. An extreme value doesn't always mean a bad answer, so read them rather than assuming.
Classifier sampling
Use an existing eval or a cheap model to flag likely failures, like answers that mention a competitor or prices far below list. It's efficient, and it mostly finds problems the classifier already knows about.
Feedback sampling
Pull traces with a thumbs down, a complaint, a retry or a handoff to a human. It's the cheapest signal you have, and it misses every failure users didn't notice, which in some domains are the worst ones.
Active learning
Once you have labels, ask your expert to label the traces that would help most next: clusters with few labels, or traces where your LLM judge is unsure. Each label then adds more information than a random one would.
Mix by stage
| Stage | Mix |
|---|---|
| First weeks, failure modes unknown | Mostly random and clustered |
| Failure modes known, judges built | More classifier-flagged and outliers, keep a random slice |
| Mature product | Targeted signals plus a steady random slice and new clusters |
Write the recipe down
import random
def build_batch(traces: list[dict], size: int = 100, seed: int = 7) -> list[dict]:
rng = random.Random(seed)
flagged = [t for t in traces if t["thumbs_down"] or t["retries"] > 1]
slow = sorted(traces, key=lambda t: t["latency_s"], reverse=True)[: size // 5]
picked = {t["id"]: t for t in rng.sample(flagged, min(len(flagged), size * 3 // 10))}
picked.update({t["id"]: t for t in slow})
rest = [t for t in traces if t["id"] not in picked]
picked.update({t["id"]: t for t in rng.sample(rest, size - len(picked))})
return list(picked.values())
rng = random.Random(1)
traces = [
{"id": i, "thumbs_down": rng.random() < 0.05, "retries": rng.choice([0, 0, 0, 1, 2]), "latency_s": rng.uniform(0.5, 9)}
for i in range(10_000)
]
batch = build_batch(traces)
print(len(batch), "traces in this week's batch")
print(sum(t["thumbs_down"] for t in batch), "with a thumbs down")
A written recipe can be repeated next sprint and improved, which a "let's look at some traces" habit can't.
A real incident
In December 2023, a Chevrolet dealership's chatbot was talked into "agreeing with anything" and then agreed to sell a 2024 Tahoe for $1, calling it "a legally binding offer". A random sample of normal sales chats would almost never contain that conversation. A sampler that flags unusual instructions to the bot, or prices far below list, would.
From the floor
I haven't run a trace-sampling exercise in a session yet, so this play comes from the playbook research.
Related
- Find failure modes in traces
- Building an eval set from production
- One domain expert
- Observability and tracing