Chaos Engineering — Resilience Patterns

Chaos engineering runs controlled experiments: define steady-state metrics (e.g., checkout success rate), form a hypothesis ("if the recommendations service is down, checkout success stays ≥ 99.9%"), inject a fault (kill instances, add latency, drop packets, fail a dependency), observe, and fix what breaks. Start in staging with small blast radius, add abort conditions, then run carefully in production. Tools: AWS Fault Injection Service, Gremlin, Chaos Mesh, LitmusChaos, Toxiproxy.

Fault types and tools

Fault	Tool examples
Instance/pod termination	AWS FIS, Chaos Mesh, Litmus
Network latency / loss	Toxiproxy, tc netem, Chaos Mesh
Dependency errors	Service mesh fault injection (Istio)
Resource exhaustion	stress-ng, Gremlin
AZ/region failure	AWS FIS scenarios, DNS failover drills

Experiment: recommendations latency
Steady state: checkout success ≥ 99.9%, product page p99 < 600 ms
Hypothesis: with +2 s latency on recommendations, product pages stay < 600 ms (breaker + fallback)
Method: Toxiproxy latency toxic on recommendations for 10 minutes, 10% of pods in staging
Abort if: checkout success < 99% or error budget burn > 5×
Result: p99 = 2.4 s → fallback missing timeout → fix: 200 ms timeout + breaker

Common pitfalls

Chaos without hypotheses or abort conditions.
Starting in production with large blast radius.
Not fixing what experiments reveal.

Key points

Hypothesis-driven experiments
Small blast radius + abort conditions
Fix findings

Further reading: Principles of Chaos Engineering

Practice questions

What makes a chaos experiment different from randomly breaking things?

Answer

It has a defined steady state, a hypothesis, controlled blast radius, abort conditions and measured outcomes that drive fixes.

Why define abort conditions?

Answer

To stop the experiment automatically if customer impact exceeds acceptable limits.