Sign in Get started

Idempotency & Retry-Safe Systems · Capstone

Capstone: Postmortem of a Retry Storm

Analyze an outage made worse by retries and fix the system.

Scenario

At 14:02 the inventory database slowed down (p99 2 s). Checkout → cart → pricing → inventory each retried 3 times with 1 s fixed delays and 30 s timeouts. Load on inventory rose 40×, the database CPU hit 100% and the outage lasted 75 minutes even after the original slow query was fixed at 14:10.

Requirements

  • Explain why the outage persisted
  • Propose fixes at each layer
  • Define how to verify the fixes

Constraints

  • No big-bang rewrite; changes must ship incrementally

Deliverables

  1. Explain the failure dynamics.
  2. Propose fixes and how to verify them.

Draft each deliverable below before revealing the model answer and rubric.

Further reading: Google SRE: Cascading failures

Deliverables

Explain the failure dynamics.

Model answer

Retry amplification: 3 retries at 4 layers multiplies attempts (up to 3^4 = 81×); fixed 1 s delays synchronized retries into waves; 30 s timeouts kept threads/connections busy, exhausting pools upstream; after the trigger was removed, the retry load alone kept inventory saturated — a metastable failure.

Rubric

Strong answers name amplification, synchronization, resource exhaustion from long timeouts, and metastability (the system stays down after the trigger ends).

Propose fixes and how to verify them.

Model answer

  • Retry only at the edge (checkout) or at the call nearest the dependency; others fail fast
  • Exponential backoff with full jitter, max 2 retries, retry budget ≤10%
  • Timeouts from latency budgets (e.g., inventory 300 ms), overall request deadline propagated
  • Circuit breaker on inventory with fallback (show 'availability unknown', allow order with reservation later)
  • Load shedding at inventory (concurrency limit, fast 503 + Retry-After)
  • Verify: load test + fault injection (latency on inventory DB) in staging; game day; dashboards for retry rate and breaker state

Rubric

Look for layered fixes plus verification by fault injection. Misses: 'add more database capacity' only.

Open in the app to mark lessons complete, take notes and track your progress.