Queues & Background Processing in Depth · Workflows and Operations
Operating Messaging Systems
Monitoring, replay, schema governance and runbooks.
Production messaging needs: dashboards for age/lag, throughput and errors per queue/consumer group; DLQ alerts with owners; replay tooling (redrive DLQs, reset consumer offsets to a timestamp); schema governance (registry with compatibility checks); tracing across async hops; capacity planning for brokers (disk, partitions, connections); and runbooks for common incidents (stuck consumer, poison message, broker failover, lag exceeding retention).
Operational tooling
| Concern | Tooling |
|---|---|
| Lag/age dashboards | Prometheus exporters, CloudWatch, Burrow |
| Schema governance | Confluent/Apicurio schema registry |
| Tracing async hops | OpenTelemetry context in headers |
| Replay | DLQ redrive, offset reset, event store |
| Broker capacity | Disk, partitions, connections monitoring |
Runbook: consumer group 'fulfillment' lag rising
1. Check consumer instances healthy? (crash loops, rebalancing storms)
2. Check processing errors / DLT rate (poison message?)
3. Check downstream latency (DB/API slow?) — handler p99
4. Scale consumers up to partition count; enable parallel per-key processing if needed
5. If lag approaches retention: extend retention.ms temporarily
6. After fix: monitor lag → 0; replay DLT messages at controlled rate
# Reset offsets (consumer group stopped):
kafka-consumer-groups --bootstrap-server $B --group fulfillment --topic orders --reset-offsets --to-datetime 2026-10-11T09:00:00.000 --execute
Common pitfalls
- No owner per queue/topic.
- Offset resets without understanding idempotency impact.
- Schema changes deployed without compatibility checks.
Key points
- Dashboards + owned alerts
- Replay tooling ready before incidents
- Schema registry + tracing
Further reading: Confluent: Schema evolution and compatibility
Practice questions
What must be true before resetting a consumer group's offsets to replay events?
Answer
Consumers must be idempotent (replayed events will be reprocessed), the group must be stopped, and you should understand the downstream impact of reprocessing.
What does a schema registry's compatibility mode do?
Answer
It rejects schema changes that would break existing producers or consumers (backward/forward/full compatibility), preventing incompatible events from being published.
Open in the app to mark lessons complete, take notes and track your progress.