How to evaluate an AI agent end to end and step by step
Start by treating the agent as a black box and ask one question for each task: did it achieve the user's goal, checked against the real final state (the database, the ticket, the file), not against what the agent says it did. Agents are random, so run each task several times and report pass^k, the chance that all k tries succeed, because that's what users feel. Once you know which tasks fail, open the box and score each step: tool choice, arguments, handling of failed tool results, permissions and cost.
Words for agent evals
Anthropic's agent eval guide gives a useful vocabulary:
- Task: one test, with an input and a success rule.
- Trial: one attempt at a task.
- Transcript: the full record of a trial.
- Outcome: the final state of the system after the trial.
- Grader: the code or model that scores it.
Write the success rule down before you run anything. "The order is cancelled in the database and the customer received a confirmation" is a success rule. "The agent handled it well" is not.
End to end: check the real outcome
Check the outcome, not the agent's final message. If the task was cancelling an order, query the database. Agents often report success they didn't achieve: the tool call failed, the agent said "Done!".
Run each task several times
The same agent on the same task can succeed once and fail the next time. So measure over several trials:
- pass@k: the chance that at least one of k tries works. Useful when a person picks the best of several attempts.
- pass^k: the chance that all k tries work. That's what users feel, because each user gets one try, and many users get the task.
def pass_at_k(p: float, k: int) -> float:
return 1 - (1 - p) ** k
def pass_hat_k(p: float, k: int) -> float:
return p ** k
p = 0.75
for k in (1, 3, 8):
print(f"k={k}: pass@k {pass_at_k(p, k):.0%} pass^k {pass_hat_k(p, k):.0%}")
Anthropic's example: 75% success per try becomes about 42% when all three tries must succeed. The τ-bench paper (2024) found GPT-4o agents succeeding on under 50% of tasks in its retail domain, and under 25% at pass^8.
Step by step: open the box
Once you know which workflows fail, score each step.
- Tool choice: did it call the right tool, or skip one it needed?
- Arguments: were the arguments correct and valid?
- Results: did it handle empty results, errors and timeouts, or plough on?
- Memory: did it remember earlier instructions and facts from the conversation?
- Efficiency: how many steps, tokens and how much money did it use?
Check each tool call as four separate things: the tool name, the arguments, the result and the change it made. Use plain code for these, not an LLM judge, because they have exact answers.
Test permissions
A well-formed tool call is still a failure if the user never approved it. Add tasks where the right behaviour is to stop and ask: refunds above a limit, deleting data, sending messages to customers. See human in the loop.
When everything fails, suspect the task
If a top model scores 0% on a task across many tries, suspect the task or the grader first. Anthropic saw a CORE-Bench score go from 42% to about 95% after fixing grading and task bugs.
A real incident
In the Replit incident (July 2025), the user's goal was "keep working, but don't touch production during the freeze". An outcome check (is the production database intact?) fails instantly. A permission check on each step (was a destructive command run without approval?) points to the exact failing step. Replit's fix was to separate development and production databases and add a planning-only mode.
From the floor
In Agent Forge, our multi-agent system with planner, coder, reviewer and tester agents, the production version had a metrics configuration for judging the system, test cases, and Langfuse for tracing every step, and I made the point that evaluation is different for agents than for a single LLM call, and different again for each domain, so for a software engineering agent that generates code, the tests are part of the evaluation.
In the agent demo, when the tester agent found an issue in the generated code, the system stopped and offered options to replan, stop or retry with a hint from the user, which is a step-level checkpoint a person can act on.