Idempotency & Retry-Safe Systems · Idempotency in Messaging and Data Pipelines

Idempotent Consumers and the Inbox Pattern

Process each message's effect once despite redelivery.

Brokers deliver at least once, so consumers must absorb duplicates. The inbox pattern records each processed message ID in the consumer's database in the same transaction as its side effects; a duplicate hits the unique constraint and is skipped. Alternatives: naturally idempotent handlers (upserts, absolute state) or dedupe caches with TTL for low-stakes work. Choose a stable message ID (event ID from the producer, not broker offsets that change on replay).

Dedupe options for consumers

Approach Guarantee Cost
Inbox table (same tx) Strong One insert per message; cleanup
Natural idempotency Strong (if truly absolute) Design effort
Redis dedupe set with TTL Best effort Fast; not transactional
Broker dedupe (e.g., SQS FIFO 5-min window) Within window only Limited window
async def handle(event: dict) -> None:
    async with db.transaction():
        inserted = await db.fetchval(
            "INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING 1",
            event["id"])
        if not inserted:
            return                       # duplicate: already applied
        await apply_payment(db, event)   # side effects in the same transaction
# ack the message only after the transaction commits

Common pitfalls

  • Dedupe keyed by broker offset or delivery tag.
  • Dedupe record and side effect in separate transactions.
  • Unbounded inbox tables (partition or expire).

Key points

  • Stable event IDs
  • Inbox insert + effect in one transaction
  • Ack after commit

Further reading: microservices.io: Idempotent consumer

Practice questions

Why shouldn't you dedupe by Kafka offset?

Answer

Offsets identify positions, not business events; replays, re-publishing or topic migrations give the same event a different offset (and different events can't be compared across topics).

What's the limitation of SQS FIFO deduplication?

Answer

It only deduplicates within a 5-minute window and only for messages sent with the same deduplication ID; longer-delayed duplicates still need consumer idempotency.

Open in the app to mark lessons complete, take notes and track your progress.