How to find your LLM app's failure modes in 30 to 100 traces
Error analysis means reading real traces and writing down what went wrong. Pull around 100 varied traces, write one plain note on the first failure in each, group the notes into a handful of named failure modes, and count them. The counts are your priority list: fix the easy ones directly, and build automated evals only for the failures that keep coming back or cost a lot.
Why start by reading
Every good eval starts here, because reading shows you the failures that actually happen, not the ones you imagined in a planning meeting. Teams that skip it end up measuring generic things like "helpfulness" while users hit a date bug nobody wrote a check for.
Step 1: pull the traces
Pull around 100 varied traces. Mix random ones with ones that show warning signs, like a thumbs down, a retry or a handoff to a human. See how to pick which traces to read.
Label at least 30 of them yourself, by hand, before you let any AI tool suggest categories. If a tool proposes categories first, you'll see your traces through its categories.
Step 2: open coding
For each trace, write one plain note about the first thing that went wrong. For example: "said the tour was at 3pm but the calendar showed 4pm", or "answered from the 2023 policy, the 2025 one was in the context".
Only note the first failure. Errors pile up: if retrieval pulled the wrong document, the wrong answer and the wrong follow-up are all the same root cause, and noting all three would make one problem look like three.
If a trace has no failure, mark it as a pass. You need those too, to know your real pass rate.
Step 3: axial coding
Read all your notes together and group them into a few named failure modes, like "date math wrong", "missed handoff to human" or "used outdated policy". This method comes from grounded theory, a qualitative research method: open coding labels individual observations, axial coding finds the categories that connect them.
Good failure mode names are specific enough that two people would put the same trace in the same group.
Step 4: keep reading until nothing new appears
Keep reading past 100 traces while you're still finding new failure modes. Stop when new traces stop teaching you anything. Researchers call this saturation.
Step 5: count
Count each failure mode in a spreadsheet or a few lines of code.
from collections import Counter
notes = [
("t01", "date math wrong"),
("t02", "pass"),
("t03", "missed handoff to human"),
("t04", "date math wrong"),
("t05", "used outdated policy"),
("t06", "date math wrong"),
("t07", "pass"),
("t08", "missed handoff to human"),
]
counts = Counter(mode for _, mode in notes if mode != "pass")
failures = sum(counts.values())
print(f"pass rate: {1 - failures / len(notes):.0%}")
for mode, n in counts.most_common():
print(f"{mode:<26}{n} ({n / failures:.0%} of failures)")
The counts are your priority list, and three or four failure modes usually cause most of the pain.
Step 6: fix or build an eval
- If the fix is obvious and cheap (a missing instruction, a bug in the date parser), fix it right away.
- If the failure keeps coming back, or costs a lot when it happens, write an automated eval for it: a code check where possible, an LLM judge where it isn't.
Not every failure needs an eval. An eval is a maintenance cost, so spend it on the failures that matter.
Step 7: repeat on a schedule
Your product, your users and your models change, so your failure modes change too. Husain and Shankar report cycles of 2 to 4 weeks, with 10 to 20 traces reviewed each week in between.
A real example
NurtureBoss, an AI assistant for apartment leasing, ran this process. Three failure types (conversation flow, handoffs and rescheduling) caused over 60% of problems. Date handling failed 66% of the time on requests like "two weeks from now". After targeted fixes, date handling success went from 33% to 95%. These figures come from Hamel Husain's field guide and haven't been independently checked.
From the floor
When we built DevFix, a builder asked whether the agent could fix any Python function, whoever wrote it, by working through the error logs, and that is the same habit at a smaller scale: read what actually failed, find the first error, and fix that before anything else.
Related
- How to pick which traces to read
- One domain expert
- Pass or fail vs a 1 to 5 score
- What a trace records