Distributed Systems for Backend Engineers · Coordination
Service Discovery, Membership and Health
Find healthy instances in a changing fleet.
Instances come and go. Service discovery maps service names to healthy endpoints: DNS (Kubernetes Services), registries (Consul, Eureka), or service meshes. Membership/failure detection uses heartbeats or gossip protocols (SWIM in Consul/memberlist). Health checks must reflect real ability to serve (readiness), and clients should use outlier detection to eject instances that fail even when health checks pass (gray failures).
Discovery and membership
| Mechanism | How | Examples |
|---|---|---|
| DNS-based | Service name → instance IPs | Kubernetes Services, Cloud Map |
| Registry | Instances register + heartbeat | Consul, Eureka |
| Gossip membership | Nodes exchange liveness info | Consul (Serf/SWIM), Cassandra |
| Mesh/xDS | Control plane pushes endpoints | Istio, Envoy, Linkerd |
| Outlier detection | Eject endpoints with errors | Envoy outlier_detection |
# Envoy outlier detection: eject hosts returning consecutive 5xx (gray failures)
outlier_detection:
consecutive_5xx: 5
interval: 10s
base_ejection_time: 30s
max_ejection_percent: 50
Common pitfalls
- Long DNS TTLs routing traffic to dead instances.
- Health checks that only test process liveness.
- Ejecting too many hosts during widespread issues (cap ejection %).
Key points
- Discovery maps names to healthy endpoints
- Gossip/heartbeats detect failures
- Outlier detection for gray failures
Further reading: Envoy: Outlier detection
Practice questions
What does outlier detection catch that health checks miss?
Answer
Instances that pass health checks but fail real requests (gray failures); clients observe errors and eject them temporarily.
Why cap the maximum ejection percentage?
Answer
During a widespread issue, ejecting most hosts would overload the remaining ones; the cap keeps enough capacity serving.
Open in the app to mark lessons complete, take notes and track your progress.