Queues & Background Processing in Depth · Messaging Foundations
Choosing a Message Broker
Kafka, RabbitMQ, SQS/SNS, Google Pub/Sub, Redis Streams and NATS.
Choose by workload: Kafka (and compatible Redpanda/MSK) for high-throughput event streams with replay and long retention; RabbitMQ for flexible routing, per-message acks, priorities and classic job queues; SQS/SNS or Google Pub/Sub for fully managed queues/fan-out with minimal ops; Redis Streams for lightweight streams when you already run Redis; NATS/JetStream for low-latency messaging and edge/IoT. Operational capacity often matters more than features.
Broker comparison
| Criterion | Kafka | RabbitMQ | SQS/SNS | Redis Streams |
|---|---|---|---|---|
| Model | Partitioned log | Queues + exchanges | Managed queue + fan-out | Log in Redis |
| Replay | Yes (retention) | Limited (streams plugin) | No | Yes (until trimmed) |
| Throughput | Very high | High | High (managed scaling) | High (memory-bound) |
| Routing | Topics/keys | Rich (direct/topic/headers) | SNS filters | Basic |
| Priorities / delays | No | Yes | Delays (≤15 min) | Manual |
| Ops burden | High (unless managed) | Medium | None | Low (if Redis exists) |
Common pitfalls
- Running self-managed Kafka without the expertise.
- Choosing by hype rather than access patterns.
- Using Redis Streams without persistence configured.
Key points
- Streams vs queues vs managed
- Match features to workload
- Ops capacity is a real constraint
Further reading: Kafka vs RabbitMQ overview
Practice questions
When would you pick SQS over Kafka?
Answer
For job queues and decoupling where replay isn't needed and you want zero operational overhead and automatic scaling.
What RabbitMQ features make it suited to task queues?
Answer
Per-message acknowledgements, prefetch control, dead-letter exchanges, priorities, delayed/TTL routing and flexible exchange routing.
Open in the app to mark lessons complete, take notes and track your progress.