Reconciliation and Repair — Idempotency & Retry-Safe Systems

Even with idempotency, systems drift: provider timeouts with unknown outcomes, bugs, manual fixes, partial outages. Reconciliation jobs periodically compare sources of truth — your ledger vs the payment provider's settlement report, orders vs shipments, outbox vs consumer inbox — and flag or auto-repair differences. For unknown outcomes, query the provider by your reference before retrying. Make repair actions idempotent and audited.

Common reconciliations

Discrepancy	Detection	Repair
Charge succeeded, order not marked paid	Provider report vs orders	Mark paid (idempotent), send receipt
Order paid, no charge at provider	Ledger vs provider	Investigate; refund/cancel
Event published, consumer never applied	Outbox vs inbox counts/ids	Replay event (idempotent consumer)
Duplicate charge	Provider charges per idempotency key/order	Refund duplicate, alert

-- Orders paid in our system without a matching provider capture (daily job)
SELECT o.id, o.total_cents
FROM orders o
LEFT JOIN provider_settlements s ON s.order_ref = o.id
WHERE o.status = 'paid' AND o.paid_at::date = current_date - 1 AND s.order_ref IS NULL;

Common pitfalls

No reconciliation until finance notices.
Automatic repairs without idempotency or audit logs.
Ignoring unknown-outcome cases instead of querying the provider.

Key points

Reconcile against sources of truth
Query by reference for unknown outcomes
Idempotent, audited repairs

Further reading: Stripe: Payout reconciliation

Practice questions

What should you do when a payment call times out?

Answer

Don't blindly retry with a new key: retry with the same idempotency key, or look the payment up by your own reference to learn the actual outcome.

Why must reconciliation repairs be idempotent?

Answer

Reconciliation jobs rerun and can overlap; repeating a repair must not double-refund or double-ship.