Messaging, Events and Email Systems · Event-Driven Architecture
Operating Queues and Streams
Dead-letter queues, backpressure, monitoring and replay.
Production messaging needs: dead-letter queues for messages that fail repeatedly (with alerts and tooling to inspect and replay), retry policies with backoff, backpressure (bounded queues, consumer concurrency limits, autoscaling on lag), and monitoring of consumer lag / oldest message age, throughput and error rates. Document runbooks for replays and poison messages.
Alerts
- queue.oldest_message_age > 5 min (orders) → consumers stuck or under-provisioned
- kafka.consumer_lag{group="fulfillment"} rising for 15 min
- dlq.depth > 0 (payments) → investigate poison messages
Autoscale workers on lag/age (e.g., KEDA ScaledObject on SQS queue length)
Replay: fix bug → move DLQ messages back to the main queue in batches → watch error rate
Common pitfalls
- DLQs nobody looks at.
- Infinite retries blocking partitions.
- Replaying DLQs before fixing the bug.
Key points
- DLQ + alerts + replay tooling
- Backpressure and autoscaling on lag
- Monitor age, lag, errors
Further reading: KEDA scalers
Practice questions
Why alert on DLQ depth?
Answer
Messages there represent failed business work (orders, payments) that won't complete without intervention.
What is backpressure in a messaging system?
Answer
Mechanisms that slow producers or limit intake when consumers can't keep up, preventing unbounded growth and resource exhaustion.
Open in the app to mark lessons complete, take notes and track your progress.