Free course · 20 lessons · Intermediate
Idempotency & Retry-Safe Systems
Make every operation safe to retry: idempotent design, Idempotency-Key implementation, timeouts and deadlines, backoff with jitter, retry budgets, idempotent consumers, outbox, exactly-once in practice and reconciliation.
1. Why Retries Happen and What Idempotency Means
Failure modes that force retries, and how to design operations that tolerate them.
- Why Retries Are Inevitable — Timeouts, unknown outcomes and the two generals problem in everyday APIs.
- Idempotency, Safety and Natural Idempotency — Precise definitions and how to make operations naturally repeatable.
- At-Most-Once, At-Least-Once and Effectively-Once — Delivery guarantees across HTTP calls and message brokers.
- Designing Idempotent Writes — Patterns: absolute updates, conditional writes, upserts and state machines.
- HTTP Semantics and Retry Policies — Which requests clients may retry automatically, and how servers signal retry behavior.
2. Idempotency Keys in Practice
Implement the Idempotency-Key pattern correctly under concurrency and failures.
- The Idempotency-Key Pattern — How clients and servers use keys to deduplicate unsafe requests.
- Storing Keys: Database vs Redis — Where to keep keys and responses, and how to make storage atomic with the work.
- Request Fingerprints and Key Misuse — Detect reuse of a key with a different request.
- Concurrent Duplicates and Locking — Handle two identical requests arriving at the same time.
- Expiry, Partial Failures and Recovery Points — Clean up keys and resume operations that failed halfway.
3. Retries Done Right
Timeouts, deadlines, backoff, jitter and retry budgets that don't amplify outages.
- Timeouts and Deadline Propagation — Set timeouts from latency budgets and pass deadlines downstream.
- Exponential Backoff and Jitter — Space out retries so recovering systems aren't flattened.
- Retry Budgets and Retry Amplification — Prevent retries at every layer from multiplying load during outages.
- Retry Policies for Clients and SDKs — Ship safe defaults in your SDKs and document retry behavior for API consumers.
4. Idempotency in Messaging and Data Pipelines
Inbox/outbox, exactly-once claims and reconciliation.
- Idempotent Consumers and the Inbox Pattern — Process each message's effect once despite redelivery.
- The Outbox Pattern and Dual Writes — Publish events reliably alongside database changes.
- Exactly-Once in Practice — What Kafka EOS, transactional sinks and workflow engines really guarantee.
- Reconciliation and Repair — Detect and fix the inconsistencies that idempotency can't prevent.
5. Capstone
Design a retry-safe payments API and analyze a retry storm.
- Capstone: Design a Retry-Safe Payments API — Make payments safe under client retries, timeouts and crashes.
- Capstone: Postmortem of a Retry Storm — Analyze an outage made worse by retries and fix the system.