Idempotency & Retry-Safe Systems · Idempotency in Messaging and Data Pipelines
Idempotent Consumers and the Inbox Pattern
Process each message's effect once despite redelivery.
Brokers deliver at least once, so consumers must absorb duplicates. The inbox pattern records each processed message ID in the consumer's database in the same transaction as its side effects; a duplicate hits the unique constraint and is skipped. Alternatives: naturally idempotent handlers (upserts, absolute state) or dedupe caches with TTL for low-stakes work. Choose a stable message ID (event ID from the producer, not broker offsets that change on replay).
Dedupe options for consumers
| Approach | Guarantee | Cost |
|---|---|---|
| Inbox table (same tx) | Strong | One insert per message; cleanup |
| Natural idempotency | Strong (if truly absolute) | Design effort |
| Redis dedupe set with TTL | Best effort | Fast; not transactional |
| Broker dedupe (e.g., SQS FIFO 5-min window) | Within window only | Limited window |
async def handle(event: dict) -> None:
async with db.transaction():
inserted = await db.fetchval(
"INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING 1",
event["id"])
if not inserted:
return # duplicate: already applied
await apply_payment(db, event) # side effects in the same transaction
# ack the message only after the transaction commits
@KafkaListener(topics = "payments.captured")
@Transactional
public void on(PaymentCaptured event) {
if (inbox.markProcessed(event.eventId()) == 0) return; // INSERT ... ON CONFLICT DO NOTHING
ledger.recordCapture(event);
}
export async function handle(event: PaymentCaptured) {
await db.$transaction(async (tx) => {
const r = await tx.$executeRaw`INSERT INTO processed_events (event_id) VALUES (${event.id}) ON CONFLICT DO NOTHING`;
if (r === 0) return; // duplicate
await tx.ledgerEntry.create({ data: toEntry(event) });
});
}
Common pitfalls
- Dedupe keyed by broker offset or delivery tag.
- Dedupe record and side effect in separate transactions.
- Unbounded inbox tables (partition or expire).
Key points
- Stable event IDs
- Inbox insert + effect in one transaction
- Ack after commit
Further reading: microservices.io: Idempotent consumer
Practice questions
Why shouldn't you dedupe by Kafka offset?
Answer
Offsets identify positions, not business events; replays, re-publishing or topic migrations give the same event a different offset (and different events can't be compared across topics).
What's the limitation of SQS FIFO deduplication?
Answer
It only deduplicates within a 5-minute window and only for messages sent with the same deduplication ID; longer-delayed duplicates still need consumer idempotency.
Open in the app to mark lessons complete, take notes and track your progress.