Queues & Background Processing in Depth · Messaging Foundations
Queues, Pub/Sub and Streams
Three messaging models and what each is for.
Point-to-point queues distribute work: each message goes to one of many competing consumers and is deleted when acknowledged (SQS, RabbitMQ queues, Celery/BullMQ). Publish/subscribe fans out: every subscription gets a copy (SNS, RabbitMQ fanout/topic exchanges, Google Pub/Sub). Log-based streams keep an ordered, replayable, partitioned log; each consumer group tracks its own position (Kafka, Kinesis, Redis Streams, Pulsar). Many systems combine them: an event stream feeding per-service work queues.
Messaging models
| Criterion | Work queue | Pub/sub | Log stream |
|---|---|---|---|
| Who gets a message | One consumer | Every subscriber | Every consumer group |
| Retention | Until acked | Until delivered to subs | Time/size based, replayable |
| Ordering | Best effort (FIFO variants) | Per subscription (varies) | Per partition |
| Replay history | No | No (mostly) | Yes |
| Typical use | Background jobs | Notifications, fan-out | Event-driven integration, analytics |
Common pitfalls
- Using a work queue when several services need the same event.
- Using Kafka as a job queue with per-message acks and priorities.
- Assuming pub/sub retains messages for late subscribers.
Key points
- Queues distribute work
- Pub/sub fans out
- Streams are replayable logs
Further reading: Enterprise Integration Patterns: Messaging
Practice questions
Why is Kafka a poor fit for priority job queues?
Answer
Kafka consumes partitions in order with offset commits; it has no per-message priorities, delays or individual acknowledgements, which job queues rely on.
How can one event reach three services that each need durable work queues?
Answer
Publish to a topic/stream and give each service its own subscription or queue bound to it (e.g., SNS → three SQS queues, or Kafka consumer groups).
Open in the app to mark lessons complete, take notes and track your progress.