Distributed Systems for Backend Engineers · Distributed Systems in Practice
Testing Distributed Systems
Fault injection, Jepsen-style tests and deterministic simulation.
Unit tests miss distributed bugs. Techniques: fault injection in integration tests (Toxiproxy latency/partitions, killing containers), Jepsen-style testing (concurrent clients + partitions + checking histories for consistency violations), property-based testing of protocols, deterministic simulation testing (FoundationDB, TigerBeetle, Antithesis — run the whole system in a simulated, reproducible environment with injected faults), and model checking (TLA+) for critical protocols.
Testing approaches
| Technique | Finds | Effort |
|---|---|---|
| Fault injection (Toxiproxy, chaos) | Timeout/retry/failover bugs | Low–medium |
| Jepsen-style history checking | Consistency anomalies | High |
| Deterministic simulation | Rare interleavings, reproducibly | High upfront |
| TLA+ model checking | Design-level protocol bugs | Medium (specialized skill) |
| Property-based tests | Edge cases in logic | Medium |
Toxiproxy in an integration test (concept)
proxy = toxiproxy.create("postgres", listen=":15432", upstream="postgres:5432")
proxy.add_toxic("latency", latency=2000) # simulate slow DB
run_checkout_test() # assert timeouts + fallback behave
proxy.disable() # simulate partition
assert_eventually(lambda: orders_consistent())
Common pitfalls
- Only testing happy paths with all services healthy.
- Flaky distributed tests without reproducibility.
- Skipping design review of protocols.
Key points
- Inject faults in tests
- Check histories for consistency
- Simulation/TLA+ for critical cores
Further reading: Jepsen
Practice questions
What does Jepsen-style testing check?
Answer
It records histories of concurrent client operations during injected faults (partitions, crashes, clock skew) and verifies them against a consistency model like linearizability.
What's the advantage of deterministic simulation testing?
Answer
Faults and scheduling are controlled by a seed, so rare interleavings found in simulation can be replayed exactly to debug them.
Open in the app to mark lessons complete, take notes and track your progress.