How a transition failure matrix finds the broken step in an agent
A transition failure matrix is a grid where each row is the last step that worked and each column is the step where the first failure happened. Every failed trace lands in exactly one cell, so counting them shows where failures cluster, which a single overall failure rate can't do. Start your debugging with the busiest cell.
Why one failure rate isn't enough
"Our agent fails 18% of tasks" tells you something is wrong. It doesn't tell you whether the planner chose the wrong tool, the SQL writer produced invalid queries, or the formatter dropped the result. Teams in that position usually start editing the planner prompt, because it feels like the brain of the agent, and often they're editing the wrong step.
Building the matrix
- Name the steps in your agent's workflow. For a text-to-SQL agent: ParseRequest, DecideTool, GenSQL, ExecSQL, FormatAnswer.
- For each failed trace, record the last step that worked and the first step that broke.
- Each failed trace lands in exactly one cell. Count them.
from collections import Counter
steps = ["ParseRequest", "DecideTool", "GenSQL", "ExecSQL", "FormatAnswer"]
failed = [
("GenSQL", "ExecSQL"), ("GenSQL", "ExecSQL"), ("DecideTool", "GenSQL"),
("GenSQL", "ExecSQL"), ("ExecSQL", "FormatAnswer"), ("GenSQL", "ExecSQL"),
("ParseRequest", "DecideTool"), ("GenSQL", "ExecSQL"), ("DecideTool", "GenSQL"),
]
matrix = Counter(failed)
width = max(len(s) for s in steps) + 2
print(" " * width + "".join(s[:9].ljust(11) for s in steps))
for last_ok in steps:
row = "".join(str(matrix.get((last_ok, first_bad), "")).ljust(11) for first_bad in steps)
print(last_ok.ljust(width) + row)
(last_ok, first_bad), n = matrix.most_common(1)[0]
print(f"\nstart here: {last_ok} -> {first_bad} ({n} of {len(failed)} failures)")
Rows are the last step that worked, columns the first step that broke.
Reading the matrix
In the evals FAQ's text-to-SQL example, the GenSQL to ExecSQL cell had 12 failures, while DecideTool to PlanCal had only 2. You know where to look first: the SQL being generated fails when it runs.
Most cells stay empty, because a step-by-step agent can only fail right after a step that worked. Counts in odd cells, like a failure at step 5 after step 1 worked, often mean a logging bug, or steps being skipped.
From the busiest cell to a fix
- Pull the traces from the busiest cell.
- Run a small error analysis on just those. Failures inside one step tend to look alike: a missing table join, a date format, a column name that changed.
- Turn the failure into the smallest test that still breaks, for example one request and the schema it needs.
- Fix it, and add the test to your regression suite.
Rebuild after every big change
Fixing one cell often pushes failures to the next step: once the SQL runs, the formatter may start failing on results it never saw before. Rebuild the matrix after every big change, and watch the counts move. That shift is how you know the fix worked.
Steps need to be visible
This only works if your traces record each step as its own span with a clear status. If your agent is one long loop with no named steps, start by adding step names to your traces.
A real example
The FAQ's text-to-SQL agent showed one transition, GenSQL to ExecSQL, holding six times more failures than DecideTool to PlanCal. Without the matrix, the team would have seen one failure rate and started editing the planner prompt. With it, they knew the SQL writer was the problem.
From the floor
I haven't built a transition failure matrix in a session yet. The agents we built are good candidates, because their steps already have names: DevFix's LangGraph nodes were plan, code, validate, execute, test and reflect, so each failed run can be placed in one cell of that grid.