Idempotency & Retry-Safe Systems · Idempotency Keys in Practice
Expiry, Partial Failures and Recovery Points
Clean up keys and resume operations that failed halfway.
Expire keys after the documented window (Stripe uses 24 hours) with a cleanup job or TTL. The hardest case: an operation with several steps (create order → charge card → send email) crashed halfway. Model recovery points (atomic phases) and store which phase completed; a retry with the same key resumes from the last recovery point instead of starting over. Each external call uses its own derived idempotency key ({key}:charge) so repeating a phase is safe.
Recovery-point design
| Phase type | How to make it repeatable |
|---|---|
| Local DB transaction | Atomic; record recovery point in same tx |
| External API call | Derived idempotency key per call |
| Notification | Outbox with dedupe id |
| Final response | Stored with the key for replay |
key = 01J9ZQ…
phase 1 (DB tx): create order (status=pending), record recovery_point='order_created'
phase 2: charge card with provider Idempotency-Key '01J9ZQ…:charge'
→ record recovery_point='charged', payment_id
phase 3 (DB tx): mark order paid, enqueue receipt email (outbox), recovery_point='finished', store response
Retry after crash in phase 2 → reads recovery_point='order_created' → repeats phase 2 with the same derived key
(provider dedupes the charge) → continues to phase 3.
Common pitfalls
- Keys expiring before clients stop retrying.
- Re-running all steps on retry (double side effects).
- No alerting on keys stuck in 'processing'.
Key points
- Documented expiry window
- Recovery points for multi-step operations
- Derived keys for downstream calls
Further reading: Brandur: idempotency keys and recovery points
Practice questions
Why derive a separate idempotency key for each external call?
Answer
So each external side effect is deduplicated by the provider individually; repeating a phase after a crash doesn't duplicate the charge or shipment.
What should happen to keys stuck in 'processing' long after their lease?
Answer
A sweeper should resume them from their recovery point (or mark them failed and alert), so they don't block retries indefinitely.
Open in the app to mark lessons complete, take notes and track your progress.