CodePark/ questions / Retries, timeouts & resilience
START PRACTICINGPRACTICE
Backend question · System design & architecture › Retries, timeouts & resilience

In this call chain each hop makes up to 3 attempts to the next one. The database becomes slow. What happens, and how would you redesign the timeouts and retries?

client -> API gateway -> orders-service -> pricing-service -> database
gateway:  timeout 2s, 3 attempts
orders:   timeout 5s, 3 attempts
pricing:  timeout 10s, 3 attempts

The problem is retry amplification: retries at each layer multiply. In the worst case, one client request becomes 3 x 3 x 3 = 27 database calls. This multiplies the load at the exact moment the database is struggling. A slowdown can become a full outage.

The timeouts are also inverted (in the wrong order). The gateway gives up after 2 s. But the services below it keep working for up to 10 s per attempt. So they waste resources on requests that nobody is waiting for anymore.

Redesign:

  • Retry at one layer only. Usually this is the layer closest to the failing call. The other layers fail fast.
  • Make timeouts shrink down the chain. Each callee's time budget must fit inside its caller's budget.
  • Pass a deadline down the chain. Downstream services stop working once the deadline has passed.
  • Add retry budgets. A retry budget caps retries as a share of traffic, for example at most 10% extra requests.
  • Add circuit breakers. A circuit breaker stops calling a dependency that keeps failing.
  • Use backoff with jitter. Wait longer before each retry, and add a small random extra so clients do not retry at the same moment.
  • Retries multiply at each layer, so here one request can become 27 database calls.
  • Inverted timeouts make services keep working after the caller has given up.
  • Retry at a single layer only.
  • Deadlines shrink and are passed down the chain.
  • Use retry budgets and circuit breakers.