Queues & Background Processing in Depth · Capstone
Capstone: Debug a Kafka Lag Incident
Find out why consumers fell hours behind and fix it.
Scenario
The order-events topic (12 partitions) feeds the fulfillment consumer group (6 pods). After a release, lag grew to 4 million messages over 6 hours. Pods show frequent rebalances; one partition's lag is 10× the others; handler p99 rose from 40 ms to 900 ms after a new database query was added. Retention is 24 hours.
Requirements
- Identify causes
- Immediate mitigation
- Prevent recurrence
Constraints
- Order must be preserved per order ID
- No data loss
Deliverables
- Identify the contributing causes.
- Mitigate now and prevent recurrence.
Draft each deliverable below before revealing the model answer and rubric.
Further reading: Kafka consumer configs
Deliverables
Identify the contributing causes.
Model answer
(1) Slow handler (new query, p99 900 ms) cut throughput; (2) processing time per poll exceeded max.poll.interval.ms → consumers kicked out → rebalance storms → more downtime; (3) hot partition (one large merchant keyed by merchant ID) concentrated load; (4) only 6 consumers for 12 partitions limited parallelism.
Rubric
Look for the rebalance mechanism (max.poll.interval.ms) and the hot key.
Mitigate now and prevent recurrence.
Model answer
Now: roll back or index the new query; reduce max.poll.records / raise max.poll.interval.ms; scale to 12 consumers; extend retention to 72 h temporarily to avoid loss; monitor lag drain. Prevent: key by order ID (not merchant) for even distribution while preserving per-order ordering; parallel per-key processing; performance tests on handlers; alerts on lag growth rate and rebalance frequency; lag-vs-retention alert.
Rubric
Strong answers protect against data loss (retention) first, then fix throughput and the root causes.
Open in the app to mark lessons complete, take notes and track your progress.