Sign in Get started

Idempotency & Retry-Safe Systems · Capstone

Capstone: Design a Retry-Safe Payments API

Make payments safe under client retries, timeouts and crashes.

Scenario

Your checkout service calls POST /payments on your payments API, which charges cards through an external provider and records results in Postgres. Mobile clients retry on timeouts; the provider sometimes takes 20 s; last month customers were charged twice during an incident.

Requirements

  • No double charges under any retry pattern
  • Clients can safely retry after timeouts
  • Crashes mid-payment are recoverable
  • Finance can reconcile daily

Constraints

  • p99 for successful payments < 3 s (async completion allowed beyond that)
  • Provider supports idempotency keys

Deliverables

  1. Design the API contract and idempotency behavior.
  2. Describe the server-side flow including crash recovery.
  3. How do you reconcile and alert?

Hints

  • Separate 'record intent' from 'perform side effect'.
  • Unknown outcomes are resolved by lookup, not by guessing.

Draft each deliverable below before revealing the model answer and rubric.

Further reading: Brandur: Idempotency keys

Deliverables

Design the API contract and idempotency behavior.

Model answer

  • POST /payments requires Idempotency-Key; scoped per merchant; 24 h retention
  • Responses: 201 completed (replayed on retry with Idempotent-Replayed: true), 202 + /payments/{id} when still processing after 3 s, 409 + Retry-After for concurrent in-progress duplicates, 422 for key reuse with a different body
  • GET /payments/{id} and GET /payments?reference= to resolve unknown outcomes

Rubric

Look for: required key, scope, replay semantics, 409/422 cases, async path for slow provider, lookup by reference. Misses: optional keys, new key per retry, no way to learn the outcome.

Describe the server-side flow including crash recovery.

Model answer

  1. Tx: claim key (INSERT … ON CONFLICT DO NOTHING, lease 60 s) + create payment(status=pending) + recovery_point=created
  2. Call provider with derived key {key}:charge; on success tx: payment=succeeded, store response, recovery_point=finished; enqueue receipt via outbox
  3. Timeout: keep status=pending, return 202; a worker resumes from the recovery point with the same derived key (provider dedupes)
  4. Sweeper resumes keys whose lease expired

Rubric

Strong answers make every phase repeatable and use the provider's idempotency. Misses: charging before recording intent, retrying with a fresh provider key.

How do you reconcile and alert?

Model answer

Daily job compares payments (succeeded/pending) with provider settlement reports by reference: succeeded-at-provider-but-pending locally → mark succeeded; captured twice → refund + incident; alerts on keys stuck in processing > 10 min and on reconciliation mismatches.

Rubric

Look for comparison against the provider as source of truth and idempotent repairs.

Open in the app to mark lessons complete, take notes and track your progress.