Resilience Patterns · How Distributed Systems Fail
Timeout Budgets Across a Call Graph
Allocate latency budgets so timeouts nest correctly.
Timeouts must nest: each caller's timeout should exceed its downstream timeouts plus its own processing, and the edge request deadline caps everything. Allocate a latency budget per hop from the user-facing SLO, propagate deadlines, and make servers stop work for callers that already timed out. Misaligned timeouts (downstream longer than upstream) waste work and cause retries of operations still in progress.
Timeout mistakes
| Misconfiguration | Consequence |
|---|---|
| Downstream timeout > upstream | Work continues after caller gave up; duplicate retries |
| No timeout | Threads/connections held indefinitely |
| Same timeout everywhere | Everything times out together; no room for retries |
| Retries ignoring deadline | Requests exceed SLO anyway |
Edge SLO: p99 < 1000 ms
├─ gateway timeout 1000 ms
│ └─ checkout service 900 ms (own work ~100 ms)
│ ├─ pricing 250 ms (1 retry within budget: 2 × 120 ms)
│ ├─ inventory 250 ms
│ └─ payments (async) — (202 Accepted, completes via webhook)
Rule: child timeout × attempts + own work < parent timeout
Common pitfalls
- Default 30 s timeouts copied everywhere.
- Retries exceeding the parent's budget.
- Long synchronous calls to slow partners in the request path.
Key points
- Budgets derived from SLOs
- Child timeouts nest inside parents
- Async for slow work
Further reading: gRPC: Deadlines
Practice questions
Why must a downstream timeout be shorter than its caller's?
Answer
Otherwise the caller gives up first and may retry while the downstream is still working, duplicating load and possibly side effects.
What should you do with a dependency whose p99 exceeds your entire budget?
Answer
Take it off the synchronous path: make it asynchronous, cache its results, or degrade gracefully when it's slow.
Open in the app to mark lessons complete, take notes and track your progress.