Poison Messages and Head-of-Line Blocking — Queues & Background Processing in Depth

A poison message fails every time it's processed. In ordered systems (Kafka partitions, FIFO groups) it blocks everything behind it (head-of-line blocking). Mitigations: bounded retries then DLQ/DLT, non-blocking retry topics (retry-5s, retry-1m topics) so the main partition keeps flowing, validating schemas at the producer, and quarantining by key when order must be preserved per entity.

Handling poison messages

Strategy	Keeps flowing?	Preserves per-key order?
Retry in place forever	No (blocked)	Yes
Retry N times then DLT	After N attempts	Mostly (skips the bad message)
Non-blocking retry topics	Yes	No (retried message processed later)
Park key (quarantine)	Yes for other keys	Yes (key paused until fixed)

# Spring Kafka non-blocking retries
@RetryableTopic(attempts = "4", backoff = @Backoff(delay = 1000, multiplier = 5.0),
                dltTopicSuffix = ".DLT", autoCreateTopics = "true")
@KafkaListener(topics = "orders")
public void onOrder(OrderEvent event) { ... }
# creates orders-retry-0, orders-retry-1, ..., orders.DLT

Common pitfalls

Blocking a whole partition for one bad message.
Retry topics breaking ordering assumptions silently.
Producers sending schema-invalid messages.

Key points

Bound retries
Non-blocking retry topics when order allows
Validate at the producer

Further reading: Spring Kafka: Non-blocking retries

Practice questions

What is head-of-line blocking in a Kafka partition?

Answer

A message that keeps failing stops the consumer from advancing the partition's offset, so all later messages in that partition wait.

What's the trade-off of non-blocking retry topics?

Answer

The main flow continues, but the failed message is processed later, out of order relative to subsequent messages for the same key.