Queues & Background Processing in Depth · Messaging Foundations

Ordering Guarantees

What ordering brokers give you and how to design around it.

Global ordering doesn't scale; brokers order within a partition/queue/message group. Kafka orders per partition (same key → same partition), SQS FIFO orders per message group ID, RabbitMQ orders per queue with a single consumer. Parallel consumers, retries and redeliveries can still reorder processing. Design for it: key messages by entity, make handlers tolerate out-of-order events (version numbers, "last writer wins by version"), or carry full state instead of deltas.

Ordering by broker

Broker Ordering scope Breaks when
Kafka Per partition (key) Changing partition count; retries to other topics
SQS standard None (best effort) Always possible
SQS FIFO Per message group Group id too coarse limits throughput
RabbitMQ Per queue Multiple consumers / requeues
Google Pub/Sub Per ordering key (opt-in) Without ordering keys
async def apply_customer_update(event: dict):
    # Ignore stale events delivered out of order
    updated = await db.execute(
        '''UPDATE customers SET name = $2, email = $3, version = $4
           WHERE id = $1 AND version < $4''',
        event["customer_id"], event["name"], event["email"], event["version"])
    if updated == "UPDATE 0":
        metrics.increment("events.stale_ignored")

Common pitfalls

  • Assuming global order across partitions.
  • Increasing partition count (keys move partitions).
  • Delta events applied out of order corrupt state.

Key points

  • Order is per partition/group
  • Key by entity
  • Make consumers order-tolerant

Further reading: Kafka: Concepts

Practice questions

How do you keep all events for one order in sequence in Kafka?

Answer

Use the order ID as the message key so all its events land in the same partition, and process each partition sequentially.

How can a consumer cope with out-of-order events?

Answer

Include a version or sequence number and ignore events older than the stored version, or carry full state so the latest version wins.

Open in the app to mark lessons complete, take notes and track your progress.