Why Distributed Systems Are Hard — Distributed Systems for Backend Engineers

A distributed system is one where a computer you didn't know existed can make yours unusable. Unlike a single process, components fail partially and independently; networks drop, delay, duplicate and reorder messages; clocks drift; and you can't distinguish a slow node from a dead one. The fallacies of distributed computing (the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology doesn't change, there is one administrator, transport cost is zero, the network is homogeneous) summarize the assumptions that break systems.

Fallacies and responses

Fallacy	Reality	Design response
The network is reliable	Packets drop; connections reset	Timeouts, retries, idempotency
Latency is zero	Cross-region ≈ 50–200 ms	Minimize round trips; locality
Bandwidth is infinite	Large payloads are slow/costly	Pagination, compression
The network is secure	Assume hostile networks	mTLS, zero trust
Topology doesn't change	Instances come and go	Service discovery, health checks
Clocks are synchronized	Drift and jumps happen	Logical clocks, fencing tokens

Common pitfalls

Treating remote calls like local function calls.
Assuming 'no response' means 'not executed'.
Designing for the happy path only.

Key points

Partial failure is normal
You can't tell slow from dead
Design for unreliable networks and clocks

Further reading: Designing Data-Intensive Applications (Kleppmann)

Practice questions

Why can't a node reliably tell whether another node has crashed?

Answer

A missing response could mean a crash, a network partition, a long GC pause or just slowness; timeouts are guesses, not proof.

Name three fallacies of distributed computing.

Answer

The network is reliable; latency is zero; bandwidth is infinite (also: the network is secure, topology doesn't change, one administrator, transport cost is zero, the network is homogeneous).