Retry Budgets and Retry Amplification — Idempotency & Retry-Safe Systems

If every layer of a 4-deep call chain retries 3 times, one user request can become 3^4 = 81 requests to the bottom service during an outage — a retry storm that keeps it down. Retry at one layer (usually closest to the failure or the edge), use retry budgets (retries may add at most ~10% extra load; gRPC and Envoy support this), circuit breakers that stop retries when a dependency is unhealthy, and server-side signals (Retry-After, 503) to tell clients to back off.

Retry amplification

Layers retrying (3 attempts each)	Worst-case requests at the bottom
1 layer	3
2 layers	9
3 layers	27
4 layers	81

# Envoy: cap retries to 10% of active requests per cluster (retry budget)
clusters:
  - name: inventory
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          retry_budget:
            budget_percent: { value: 10.0 }
            min_retry_concurrency: 3

Common pitfalls

SDK, service and proxy all retrying the same call.
Retries without budgets during brownouts.
Load balancers retrying on timeouts by default.

Key points

Retry at one layer
Budgets cap extra load
Breakers stop retrying dead dependencies

Further reading: Google SRE: Addressing cascading failures

Practice questions

How can retries turn a small outage into a long one?

Answer

Retries multiply load on the struggling service exactly when it has the least capacity, preventing recovery (and can trigger metastable failure states).

What is a retry budget?

Answer

A limit on retries as a fraction of normal traffic (e.g., ≤10%), so retries can absorb transient errors but can't multiply load during widespread failure.