Queues & Background Processing in Depth · Scaling Consumers
Autoscaling Workers on Queue Metrics
Scale consumers with KEDA, cloud autoscalers and serverless triggers.
CPU-based autoscaling reacts poorly to queue workloads. Scale on queue metrics: messages per worker, lag or oldest age. KEDA (Kubernetes Event-Driven Autoscaling) scales deployments on SQS, RabbitMQ, Kafka lag, Redis, Postgres queries and more, including scale-to-zero. Serverless triggers (Lambda + SQS, Cloud Run + Pub/Sub) scale automatically but need concurrency caps to protect downstream systems.
Scaling signals
| Trigger metric | Good for |
|---|---|
| Messages per replica | Uniform jobs |
| Consumer lag (Kafka) | Streams |
| Oldest message age | Latency SLOs |
| CPU | CPU-bound workers only |
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: email-workers }
spec:
scaleTargetRef: { name: email-worker }
minReplicaCount: 1
maxReplicaCount: 50 # protect the SMTP provider / DB
triggers:
- type: aws-sqs-queue
metadata:
queueURL: https://sqs.eu-west-1.amazonaws.com/123456789012/emails
queueLength: "100" # target messages per replica
awsRegion: eu-west-1
authenticationRef: { name: keda-aws }
Common pitfalls
- No max replicas (overwhelming downstream).
- Scale-down killing in-flight jobs (graceful shutdown).
- Lambda concurrency unbounded against a database.
Key points
- Scale on queue metrics
- Cap replicas for downstream safety
- Graceful shutdown on scale-in
Further reading: KEDA scalers
Practice questions
Why is CPU a poor autoscaling signal for I/O-bound queue workers?
Answer
Workers waiting on network calls use little CPU while the backlog grows, so CPU-based scaling doesn't react.
Why set a maximum replica count for queue workers?
Answer
Unbounded scaling can overwhelm databases or third-party APIs the workers call, turning a backlog into an outage.
Open in the app to mark lessons complete, take notes and track your progress.