Consume Kafka messages with retries and a dead-letter topic

Approach

Use @KafkaListener with a DefaultErrorHandler that retries with exponential backoff and then publishes the record to a dead-letter topic via DeadLetterPublishingRecoverer. Mark non-retryable exceptions (validation errors) to skip retries, and make the handler idempotent with a processed-events table.

Solution

@Configuration
class KafkaErrorHandlingConfig {
    @Bean
    DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
        var recoverer = new DeadLetterPublishingRecoverer(template,
                (record, ex) -> new TopicPartition(record.topic() + ".DLT", record.partition()));
        var backoff = new ExponentialBackOffWithMaxRetries(4);
        backoff.setInitialInterval(500);
        backoff.setMultiplier(2.0);
        var handler = new DefaultErrorHandler(recoverer, backoff);
        handler.addNotRetryableExceptions(InvalidEventException.class);
        return handler;     // Boot wires a single DefaultErrorHandler bean into the listener container factory
    }
}

@Component
class OrderEventsListener {
    private final ProcessedEventRepository processed;
    private final FulfillmentService fulfillment;
    OrderEventsListener(ProcessedEventRepository processed, FulfillmentService fulfillment) {
        this.processed = processed; this.fulfillment = fulfillment;
    }

    @KafkaListener(topics = "order.created", groupId = "fulfillment")
    @Transactional
    public void on(OrderCreated event, @Header(KafkaHeaders.RECEIVED_KEY) String key) {
        if (!processed.markProcessed(event.eventId())) return;    // INSERT ... ON CONFLICT DO NOTHING → false if seen
        fulfillment.prepareShipment(event.orderId());
    }
}

Pitfalls

• Infinite retries blocking a partition.
• Non-idempotent handlers under at-least-once delivery.
• No monitoring/alerting on the DLT.

Variations

• Non-blocking retries with @RetryableTopic (retry topics with delays).
• RabbitMQ: dead-letter exchanges with TTL-based retry queues.

Further reading: Spring for Apache Kafka: Error handling