Concurrent Duplicates and Locking — Idempotency & Retry-Safe Systems

Retries often arrive concurrently — the client timed out at 5 s while the original is still running. Claim the key atomically (INSERT … ON CONFLICT DO NOTHING, or Redis SET NX PX), and make the loser return 409 Conflict ("in progress") so the client retries later — never run the operation twice. Use a lease (locked_until) so a crashed worker's claim can be recovered.

Claiming a key

Approach	Atomic claim	Crash recovery
INSERT … ON CONFLICT DO NOTHING	Yes (unique PK)	locked_until lease + sweeper
SELECT … FOR UPDATE	Yes (row lock)	Lock released on crash
Redis SET NX PX	Yes	TTL expiry
Check-then-insert	No (race)	—

const lock = await redis.set(`idem:${scope}:${key}`, workerId, { NX: true, PX: 60_000 });
if (!lock) {
  const done = await db.idempotencyKey.findUnique({ where: { scope_key: { scope, key } } });
  if (done?.status === "completed") return res.status(done.responseStatus).json(done.responseBody);
  return res.status(409).set("Retry-After", "2").json({ title: "Request in progress" });
}

tag, err := tx.Exec(ctx, `INSERT INTO idempotency_keys (scope, key, fingerprint, status, locked_until)
    VALUES ($1, $2, $3, 'processing', now() + interval '60 seconds') ON CONFLICT DO NOTHING`, scope, key, fp)
if err != nil { return err }
if tag.RowsAffected() == 0 {
    return replayOrConflict(ctx, tx, scope, key, fp) // completed → replay; processing → 409
}

Common pitfalls

Check-then-insert races.
Holding database locks during slow external calls.
No lease: a crashed request blocks the key forever.

Key points

Atomic claim
Loser gets 409 + Retry-After
Leases for recovery

Further reading: Brandur: idempotency keys

Practice questions

Why is 'SELECT the key; if missing, INSERT it' unsafe?

Answer

Two concurrent requests can both see the key missing and both proceed, executing the operation twice; the claim must be a single atomic operation.

What does a lease (locked_until) protect against?

Answer

A worker crashing mid-request: once the lease expires, a retry can reclaim the key and finish (or resume) the operation instead of getting 409 forever.