Agent state, checkpoints and recovery
A long-running agent will meet failures: a tool times out, the model provider has a bad day, the server restarts, or the run waits hours for an approval. Keep the agent's state explicit (goal, steps done, results, pending actions), save a checkpoint after every step, make tools safe to retry, and design every failure to end in a known state: resumed, retried, escalated or stopped with an explanation. Log every step as a trace, so you can see exactly what happened in any run.
Why state matters
A demo agent runs for 20 seconds in one process, and if something fails, you run it again. A production agent may run for minutes or hours, call dozens of tools, wait for a person's approval, and run thousands of times a day.
If it crashes after step 14 of 20, starting again from step 1 wastes time and money, and worse, it may repeat actions that already happened, like sending the same email twice or refunding twice.
What state contains
Make state an explicit object, not just "whatever is in the message list".
from pydantic import BaseModel, Field
class AgentState(BaseModel):
run_id: str
goal: str
status: str = "running"
step: int = 0
plan: list[str] = Field(default_factory=list)
completed: list[dict] = Field(default_factory=list)
pending_action: dict | None = None
messages: list[dict] = Field(default_factory=list)
cost_usd: float = 0.0
With explicit state, you can save it, inspect it, resume from it, and show progress to users.
Checkpoints
After every step, save the state to durable storage, a database or a store the framework provides. If the process dies, a new process loads the last checkpoint and continues from there.
Checkpoints also make other things possible:
- Pausing for human approval and resuming later.
- Replaying a run step by step to debug it.
- Branching from a past step to try a different approach.
Frameworks like LangGraph build this in, with checkpointers backed by databases like PostgreSQL. Durable execution platforms like Temporal handle it for any long-running process.
Making tools safe to retry
Retries are only safe if repeating a tool call doesn't repeat its effect.
- Idempotency keys: pass a unique ID with each write action, and have the tool ignore a second request with the same ID.
- Check before acting: before creating a ticket, check whether one already exists for this run.
- Record side effects: store "email sent" in state as soon as the tool confirms, before the next step.
- Separate read and write tools, and retry reads freely.
Handling failures
Decide in advance what happens for each kind of failure.
| Failure | Response |
|---|---|
| Tool timeout or temporary error | Retry with backoff, then report to the model |
| Invalid tool arguments | Return the validation error to the model to correct |
| Model API error | Retry, then fall back to another model |
| Model output can't be parsed | Repair once, then stop with an error |
| No progress or repeated steps | Stop, summarise and escalate |
| Budget or step limit reached | Stop, save state, report what was done |
| Process crash | Resume from the last checkpoint |
| Waiting for approval | Save state, resume when approved |
Every run should end in a clear status: completed, failed with a reason, waiting for a person, or cancelled. A run that silently disappears is the worst outcome, because nobody knows whether its actions happened.
Timeouts and long tasks
- Give each tool call a timeout.
- Give the whole run a time budget.
- Run long agent tasks as background jobs from a queue, not inside a web request, and let users check progress or get notified.
Traces
A trace records every step of a run: the model's input and output, each tool call with arguments and results, timings, tokens and errors. It is how you debug agents, because the question is almost never "what did the code do?", it is "why did the agent decide that?". Use an LLM observability tool or your own structured logs, and keep traces long enough to investigate user reports. See LLMOps.
From the floor
In the DevFix agent, the parts I listed at the start were stateful loops, sandboxing, test passing and reflection, because an agent that writes code, runs it, reads the errors and tries again needs to know what it already tried, run that code safely, and decide from the test results whether it's done.