The concept
A slow database makes callers time out and retry; retried at every layer, one click becomes 64 attempts, and the outage outlives its cause.
A retry storm is a failure in which retries, meant to ride out a brief error, add load to a service that is already struggling, until the retries alone keep it down. It starts with a trigger, here a slow query that makes the database answer late: callers time out and retry, and each retry is more work for a database that could not keep up in the first place.
Retries multiply across layers. The Site Reliability Engineering book gives the case of a backend, a frontend and a JavaScript client each retrying 3 times: a single user action may create 64 attempts on the database. It recommends retrying at one layer only, capping a request at 3 attempts, and keeping retries under 10% of a client's requests, which bounds the extra load to about 1.1 times.
Exponential backoff makes each new attempt wait longer, and jitter makes that wait random, so callers that failed together do not come back together. Marc Brooker showed that backoff without jitter still sends calls in clusters. Neither changes how many attempts there are: they spread them over time.
A circuit breaker, as Martin Fowler describes it, sits before a call and counts failures. Past a threshold it opens, and calls fail at once without being made, which spares the struggling service; after a pause it lets a trial call through, and closes if the call succeeds. Callers get an error, or stale data, straight away instead of a long wait.
The outage can outlive its cause. Nathan Bronson and colleagues call these metastable failures: once the retries keep the load above capacity, stopping the trigger is not enough, and recovery needs a strong push, such as cutting the load well below normal. The loads, the capacity and the timeouts on this machine are modelled.