Sign in Get started

Free course · 20 lessons · Advanced

Resilience Patterns

Design systems that survive failure: failure modes and metastability, timeout budgets, circuit breakers, bulkheads, load shedding, backpressure, graceful degradation, hedging, caching for resilience, failover and chaos engineering.

Start the course

1. How Distributed Systems Fail

Cascading failures, metastability and designing from SLOs.

  1. Failure Modes of Distributed Systems — Crashes, slowness, partitions, gray failures and dependency failures.
  2. Cascading and Metastable Failures — How overload spreads and why systems stay down after the trigger ends.
  3. Timeout Budgets Across a Call Graph — Allocate latency budgets so timeouts nest correctly.
  4. SLO-Driven Resilience — Decide how much resilience you need from user-facing objectives.

2. Core Resilience Patterns

Circuit breakers, bulkheads, load shedding, backpressure, fallbacks and hedging.

  1. Circuit Breakers — Stop calling a failing dependency and let it recover.
  2. Bulkheads: Isolating Resources — Prevent one dependency or tenant from exhausting shared capacity.
  3. Load Shedding and Admission Control — Reject excess work early to keep serving the rest.
  4. Backpressure — Propagate 'slow down' signals instead of buffering until you crash.
  5. Graceful Degradation and Fallbacks — Keep the core experience working when parts fail.
  6. Hedged Requests and Tail Latency — Cut p99 latency in fan-out systems.

3. Resilient Data and Dependencies

Caching for resilience, queues, failover and dependency hygiene.

  1. Caching for Resilience — Serve stale data instead of errors when origins fail.
  2. Queue-Based Load Leveling — Absorb bursts by decoupling producers from consumers.
  3. Multi-AZ, Multi-Region and Failover — Redundancy across failure domains and how failover really works.
  4. Managing Critical and Optional Dependencies — Inventory dependencies and design each call deliberately.

4. Verifying Resilience

Chaos experiments, game days, load tests and resilience reviews.

  1. Chaos Engineering — Inject failures deliberately to find weaknesses first.
  2. Game Days and Incident Drills — Practice failures as a team before they happen for real.
  3. Load, Stress and Soak Testing — Know your limits and how the system behaves past them.
  4. Resilience Reviews and Production Readiness — A checklist to apply to every service before launch.

5. Capstone

Design a resilient checkout and fix a metastable failure.

  1. Capstone: Design a Resilient Checkout — Make checkout survive dependency failures and traffic spikes.
  2. Capstone: Recover From a Metastable Failure — Diagnose and fix a system that stays down after the trigger is gone.