Queues & Background Processing in Depth · Scaling Consumers
Consumer Concurrency and Throughput
Workers, threads, prefetch and partition-bound parallelism.
Throughput = consumers × concurrency per consumer × messages per second each. Queues (SQS, RabbitMQ) scale by adding consumers freely. Kafka parallelism is capped by partitions per consumer group; beyond that, process records concurrently inside a consumer while preserving per-key order (e.g., Confluent Parallel Consumer, key-based worker pools). I/O-bound handlers benefit from async concurrency; CPU-bound ones from more processes.
Scaling consumption
| Broker | Scale out by | Limit |
|---|---|---|
| SQS | More consumers / long polling | Effectively unlimited (standard) |
| RabbitMQ | More consumers per queue | Queue throughput (single queue master) |
| Kafka | More consumers in group | Number of partitions |
| Kafka + parallel consumer | Per-key concurrency inside consumer | Ordering per key preserved |
Throughput estimate
handler latency 50 ms (I/O bound) → 20 msg/s per worker thread
target 2,000 msg/s → 100 concurrent handlers
Kafka topic with 12 partitions → 12 consumers × ~9 concurrent per-key workers each
Common pitfalls
- Adding Kafka consumers beyond partitions (idle consumers).
- Concurrency that breaks per-key ordering.
- Downstream database becoming the real bottleneck.
Key points
- Compute required concurrency
- Kafka capped by partitions
- Watch downstream limits
Further reading: Confluent Parallel Consumer
Practice questions
Why doesn't adding consumers beyond the partition count speed up a Kafka consumer group?
Answer
Each partition is assigned to at most one consumer in the group; extra consumers sit idle.
How can you increase Kafka processing parallelism without more partitions?
Answer
Process records concurrently within each consumer, grouping by key so each key's records stay ordered (e.g., a parallel consumer library).
Open in the app to mark lessons complete, take notes and track your progress.