Reliability patterns for LLM apps: retries, fallbacks, timeouts and circuit breakers
Wrap every model and tool call with a timeout, retry transient errors with exponential backoff and jitter, fall back to a second model or provider when the first keeps failing, use a circuit breaker so you stop sending requests to a service that is down, and degrade gracefully (a cached answer, a simpler response or a handoff to a person) instead of showing an error. Reliability for LLM apps is ordinary distributed systems work plus one extra failure type: the call succeeds and the output is bad, so validation failures need a recovery path too. Evaluate your fallback model as well, because a fallback that answers badly is an outage users can't see.
What goes wrong
Your model provider is a dependency, like a payment API or a database, and dependencies fail:
- Timeouts and slow responses at peak hours.
- Rate limits (HTTP 429) when you exceed your quota.
- Server errors (500, 503) and full outages.
- Model deprecations on a schedule.
And one failure that normal APIs don't have: the call returns 200, and the output is malformed JSON, the wrong schema, a refusal, or a confident wrong answer.
Without protection, one slow provider can take down the whole app: every request waits for its full timeout, workers fill up, and features that don't even use the model stop responding.
Patterns
| Pattern | Protects against |
|---|---|
| Timeout | Hung calls holding workers |
| Retry with backoff and jitter | Short transient errors |
| Fallback model or provider | Outages, rate limits, deprecations |
| Circuit breaker | Hammering a service that's already failing |
| Output validation with repair or retry | Malformed or invalid output |
| Graceful degradation | Everything else |
| Bulkhead (separate pools and limits) | One feature starving the others |
Timeouts
Every model and tool call gets a timeout. For streaming, set one for the first token and one for the whole response. Choose values from your measured latencies, not defaults that wait minutes.
Retries with backoff and jitter
Retry only errors that can succeed on a second try: timeouts, 429, 500, 503.
Don't retry 400-type errors like an invalid request, because they'll fail the same way.
Wait longer between each try (exponential backoff), add randomness (jitter) so thousands of clients don't retry at the same moment, and cap the number of attempts.
If the provider sends a Retry-After header, respect it.
Fallbacks
When the primary model keeps failing, call a second model, ideally from a different provider. Gateways like LiteLLM can do this from configuration. Keep the fallback's prompt tested and its output validated, and run your eval set on it regularly.
Circuit breakers
A circuit breaker counts recent failures. After too many, it opens: calls fail fast (or go straight to the fallback) without waiting for timeouts. After a cool-down, it lets one test request through, and closes again if it succeeds.
Output validation
Validate structured outputs against a schema. On failure, retry once with the validation error in the prompt, then fall back or return a safe default.
Graceful degradation
When everything fails, the user should still get something useful: a cached answer for a common question, a simpler non-AI response, search results instead of a generated answer, or a handoff to a person with the context attached.
Code: retry, fallback and circuit breaker
import random
import time
class TransientError(Exception):
pass
class CircuitBreaker:
def __init__(self, threshold: int = 3, cooldown_s: float = 30.0) -> None:
self.threshold, self.cooldown_s = threshold, cooldown_s
self.failures, self.opened_at = 0, None
def allow(self) -> bool:
if self.opened_at is None:
return True
return time.monotonic() - self.opened_at > self.cooldown_s
def record(self, ok: bool) -> None:
self.failures = 0 if ok else self.failures + 1
if ok:
self.opened_at = None
elif self.failures >= self.threshold:
self.opened_at = time.monotonic()
def with_retries(call, attempts: int = 3, base_s: float = 0.2):
for attempt in range(attempts):
try:
return call()
except TransientError:
if attempt == attempts - 1:
raise
time.sleep(base_s * 2**attempt + random.uniform(0, base_s))
def answer(question: str, providers: list, breakers: dict) -> str:
for name, call in providers:
breaker = breakers[name]
if not breaker.allow():
continue
try:
result = with_retries(lambda: call(question))
breaker.record(True)
return f"[{name}] {result}"
except TransientError:
breaker.record(False)
return "Our assistant is busy right now. A team member will reply shortly."
def primary(q: str) -> str:
raise TransientError("503")
def secondary(q: str) -> str:
return "Refunds are available within 14 days in the EU."
breakers = {"primary": CircuitBreaker(), "secondary": CircuitBreaker()}
providers = [("primary", primary), ("secondary", secondary)]
for _ in range(3):
print(answer("EU refund window?", providers, breakers))
print("primary breaker open:", not breakers["primary"].allow())
Each request retries the primary three times, then falls back to the secondary. After three failed requests, the primary's breaker opens, so later requests skip it and go straight to the fallback until the cool-down ends.
Test it before you need it
Run a game day: block your primary provider in staging and watch what users would see. Check that timeouts fire, the breaker opens, the fallback answers correctly, and alerts reach the right person.
Measure:
- Availability of the AI feature.
- Fallback rate.
- Pass rate on your eval set when running on the fallback model.
From the floor
On day 1 of Build30, the first LLM client we wrote already had an HTTP timeout and retry options with 3 attempts, set when the client is created, before the first prompt was sent.
In Agent Forge's production version, the model call went through a list of fallback models from the settings: if the call to one model failed, the code moved to the next, and every response was validated against a JSON schema before it was accepted.