Incident response for AI systems
AI incidents include wrong or harmful answers at scale, leaked data, agents taking actions nobody approved, cost spikes and provider outages. Prepare before they happen: kill switches that disable a feature, a tool or a model route in one step, a triage guide by incident type, named owners, and traces good enough to see what happened. End every incident with a postmortem that produces new eval cases and guardrails, so the same failure can't reach users twice.
Companies are accountable for what their AI says
In February 2024, a Canadian tribunal held Air Canada responsible for its chatbot's wrong answer about a bereavement fare, rejecting the argument that the chatbot was responsible for its own actions. Whatever your AI says or does, your company owns it. That's why incident response for AI isn't optional.
Types of AI incidents
| Incident | Examples | First action |
|---|---|---|
| Wrong or harmful answers | Promising refunds you don't offer, unsafe advice | Disable the route or switch to a safe fallback |
| Data leak | Another customer's data in an answer, PII in logs | Disable, preserve logs, notify according to policy and law |
| Runaway agent | Deleting data, sending messages, repeated actions | Kill switch on the tool, revoke credentials |
| Cost spike | A loop, abuse, a prompt that grew | Spend cap, rate limit, find the source |
| Provider outage | Timeouts, errors | Fallback provider, status message |
| Prompt injection | The system following instructions from content | Disable the affected tool or source, review traces |
Kill switches
A kill switch turns something off in one step, without a deploy:
- Disable the AI feature and show a fallback (a form, a human handoff, search results).
- Disable one tool for an agent.
- Disable one model route and send traffic elsewhere.
- Disable one data source for retrieval.
- Lower spending limits.
Write the list for your system, put it in the runbook, and test each switch once.
Triage
- Contain: use the kill switch that stops the harm with the least collateral damage.
- Assess: how many users, which data, since when. Use traces.
- Communicate: tell the owners, support, and where required, affected users and regulators.
- Fix: the prompt, the guardrail, the tool permission, the data.
- Verify: run the evals, including a new case for this incident, before re-enabling.
Preparing
- Traces with the prompt, model and tool calls for every request, kept long enough to investigate. See observability.
- An audit log of every action agents took. See audit logs.
- Least-privilege permissions for agents, so a runaway agent can't do much. See agent permissions.
- Alerts on guardrail triggers, error rates, cost and feedback signals.
- An owner on call who can use the kill switches.
Postmortems
After every incident, write down:
- What happened, and the timeline.
- How it was detected, and how long that took.
- The root cause, with the first failure in the trace.
- What stopped it.
- New eval cases, guardrails and alerts added.
Blame the system, not people. The goal is that this failure becomes a test.
Measuring it
- Time to detect.
- Time to mitigate.
- Share of postmortems that produced a new eval case.
From the floor
In DevFix, I showed a real story where a coding agent deleted a user's 202 GB archive after being told explicitly not to remove their data, as the reason our agent had a safety guard with rule-based filters on dangerous commands and a human approval step before it moved forward.