Messaging, Events and Email Systems · Asynchronous Workflows
Queues, Streams and Background Jobs
Choose between task queues, pub/sub and event streams.
Task queues (SQS, RabbitMQ, Celery/BullMQ/Sidekiq) distribute work items to competing workers; each message is processed once and removed. Pub/sub (SNS, Google Pub/Sub, RabbitMQ fanout) delivers each event to every subscriber. Event streams (Kafka, Kinesis, Redis Streams) keep an ordered, replayable log partitioned by key; consumer groups track their own offsets. Use queues for jobs, streams for event-driven integration and replay, and pub/sub for fan-out notifications.
| Need | Choose |
|---|---|
| Send emails / resize images in background | Task queue (SQS, BullMQ) |
| Many services react to "order.paid" | Pub/sub or stream |
| Replay history, ordered per entity | Stream (Kafka) keyed by id |
| Delayed/scheduled jobs | Queue with delay support |
Common pitfalls
- Using a stream when a simple queue suffices (operational overhead).
- Using queues for events multiple services need (each message consumed once).
- Unbounded queues with no monitoring of age.
Key points
- Queues distribute work
- Pub/sub fans out
- Streams are replayable ordered logs
Further reading: Enterprise Integration Patterns
Practice questions
Why can't a standard work queue serve three independent services that all need every order event?
Answer
Each queue message is consumed by one worker; you need pub/sub fan-out (one queue per subscriber) or a stream where each consumer group reads independently.
What does Kafka's partitioning by key give you?
Answer
Ordering for all events of the same key (e.g., one order) and parallelism across keys, since partitions are consumed in parallel.
Open in the app to mark lessons complete, take notes and track your progress.