Retry storm

A slow database makes callers time out and retry; retried at every layer, one click becomes 64 attempts, and the outage outlives its cause.

Assembling the machine…

Load on the database · queries/s

30001234567

Step 1 / 7

Orders flow

250 orders a second reach a database that can take 300.

Event log · 1
  1. 20.00 msHTTPPOST /orders → API → orders service → database

The concept

A slow database makes callers time out and retry; retried at every layer, one click becomes 64 attempts, and the outage outlives its cause.

A retry storm is a failure in which retries, meant to ride out a brief error, add load to a service that is already struggling, until the retries alone keep it down. It starts with a trigger, here a slow query that makes the database answer late: callers time out and retry, and each retry is more work for a database that could not keep up in the first place.

Retries multiply across layers. The Site Reliability Engineering book gives the case of a backend, a frontend and a JavaScript client each retrying 3 times: a single user action may create 64 attempts on the database. It recommends retrying at one layer only, capping a request at 3 attempts, and keeping retries under 10% of a client's requests, which bounds the extra load to about 1.1 times.

Exponential backoff makes each new attempt wait longer, and jitter makes that wait random, so callers that failed together do not come back together. Marc Brooker showed that backoff without jitter still sends calls in clusters. Neither changes how many attempts there are: they spread them over time.

A circuit breaker, as Martin Fowler describes it, sits before a call and counts failures. Past a threshold it opens, and calls fail at once without being made, which spares the struggling service; after a pause it lets a trial call through, and closes if the call succeeds. Callers get an error, or stale data, straight away instead of a long wait.

The outage can outlive its cause. Nathan Bronson and colleagues call these metastable failures: once the retries keep the load above capacity, stopping the trigger is not enough, and recovery needs a strong push, such as cutting the load well below normal. The loads, the capacity and the timeouts on this machine are modelled.

The run, step by step

Every scenario, in words.

What the machine plays, step by step. Beside a step, what a what-if switch plays in its place, or adds right after it.

A slow query

7 steps
  1. Play

    Orders flow

    Shoppers place orders through the shop app. Each one goes through the API and the orders service to the database, which answers in 20 ms and can take 300 queries a second. It gets 250.

  2. Play

    A report slows the database

    A report job starts a heavy query on the same database. Every query now waits behind it: answers take 1.5 s instead of 20 ms.

  3. Play

    The service gives up after 1 s

    The orders service waits 1 s for the database, then gives up. The API, waiting on the service, gives up too, and so does the app. The database may still finish the query: that work is wasted.

  4. Play

    Every layer retries, at once

    The service retries its query 3 times, the API retries the service 3 times, the app retries the API 3 times. One click becomes up to 64 attempts on the database, 4 × 4 × 4, and the failed ones come back at the same moments.

    • With Retry at one layer, within a budget

      Play

      One layer retries, within a budget

      Only the orders service retries, the layer right above the database, and only while retries stay under 10% of its requests; the API and the app pass the failure on. The database sees about 1.1 times the load: 275 queries a second instead of 250.

    • With Back off with jitter

      Play

      Every layer retries, spread out

      The service retries its query 3 times, the API retries the service 3 times, the app retries the API 3 times. Each retry first waits a random time, up to a cap that doubles with every attempt, so retries stop arriving in waves. There are just as many: up to 64 attempts on the database for one click.

    • With Back off with jitter + Retry at one layer, within a budget

      Play

      One layer retries, within a budget

      Only the orders service retries, the layer right above the database, and only while retries stay under 10% of its requests; the API and the app pass the failure on. Each retry first waits a random, growing time. The database sees about 1.1 times the load: 275 queries a second instead of 250.

  5. Play

    The database drowns

    Queries arrive far faster than the slowed database answers them. Every one now times out, orders that never needed a retry included, and each timeout sends more retries.

    • With Put a circuit breaker before the database

      Play

      The breaker opens

      After 5 failures in a row, the breaker opens. Calls now fail at once, without reaching the database: shoppers get an error straight away instead of a long wait, and the database only has the report to finish. The layers above still retry, and every attempt stops at the breaker.

    • With Retry at one layer, within a budget

      Play

      Slow, but not buried

      The database is still slow while the report runs, and some orders still time out. But the load on it stays near normal: it is not buried under retries.

    • With Retry at one layer, within a budget + Put a circuit breaker before the database

      Play

      The breaker opens

      After 5 failures in a row, the breaker opens. Calls now fail at once, without reaching the database: shoppers get an error straight away instead of a long wait, and the database only has the report to finish.

  6. Play

    The cause is gone, the outage is not

    Someone stops the report, and the database is as fast as before. It stays overloaded: the retries already sent, and the ones their timeouts trigger, keep it above 300 queries a second. The retries now keep the failure going on their own.

    • With Put a circuit breaker before the database

      Play

      A trial call, then the breaker closes

      Someone stops the report. After its pause the breaker lets one trial call through: it is answered in 20 ms, so the breaker closes and calls reach the database again.

    • With Retry at one layer, within a budget

      Play

      The report stops, the database catches up

      Someone stops the report. With the load near normal, the database drains its queue and answers in 20 ms again.

    • With Retry at one layer, within a budget + Put a circuit breaker before the database

      Play

      A trial call, then the breaker closes

      Someone stops the report. After its pause the breaker lets one trial call through: it is answered in 20 ms, so the breaker closes and calls reach the database again.

    • With Back off with jitter

      Play

      The cause is gone, the outage is not

      Someone stops the report, and the database is as fast as before. It stays overloaded: the retries already sent, and the ones their timeouts trigger, keep it above 300 queries a second. The retries now keep the failure going on their own. Jitter spread them out; it did not make them fewer.

    • With Back off with jitter + Put a circuit breaker before the database

      Play

      A trial call, then the breaker closes

      Someone stops the report. After its pause the breaker lets one trial call through: it is answered in 20 ms, so the breaker closes and calls reach the database again.

    • With Back off with jitter + Retry at one layer, within a budget

      Play

      The report stops, the database catches up

      Someone stops the report. With the load near normal, the database drains its queue and answers in 20 ms again.

    • With Back off with jitter + Retry at one layer, within a budget + Put a circuit breaker before the database

      Play

      A trial call, then the breaker closes

      Someone stops the report. After its pause the breaker lets one trial call through: it is answered in 20 ms, so the breaker closes and calls reach the database again.

  7. Play

    Still down

    Getting out takes what the research on these failures calls a strong push: cut the load well below normal, or stop the retries. Until then, no order gets through.

    • With Put a circuit breaker before the database

      Play

      Orders flow again

      250 orders a second, answered in 20 ms. The fix did not stop the slow query; it kept the slowdown from turning into an outage that outlives it.

    • With Retry at one layer, within a budget

      Play

      Orders flow again

      250 orders a second, answered in 20 ms. The fix did not stop the slow query; it kept the slowdown from turning into an outage that outlives it.

    • With Retry at one layer, within a budget + Put a circuit breaker before the database

      Play

      Orders flow again

      250 orders a second, answered in 20 ms. The fix did not stop the slow query; it kept the slowdown from turning into an outage that outlives it.