Kafka Consumer Non-Blocking Retry Pattern

When an event consumed from Kafka needs to be retried, the retry can either be blocking or non-blocking. Blocking retry means that subsequent events on the partition will not be processed until the retrying event has completed successfully or otherwise. Non-blocking retry means that those later events can be consumed and processed while the original event is being retried. The trade-off for achieving non-blocking retry is that the order of events being processed can no longer be guaranteed.

This is the second of a two part series on non-blocking retry. In the first part non-blocking retry using Spring Kafka was detailed. Spring Kafka provides excellent support with minimal coding required to implement a non-blocking retry pattern. However, for applications not written using Spring, or indeed not using Apache Kafka, then this article describes the design for a generic non-blocking retry pattern that could be applied.

The source code for the accompanying Spring Boot demo application is available here.

For a general overview on consumer retry, and for use cases that require blocking retry, see the article Kafka Consumer Retry.

Non-Blocking Retry Pattern

This article covers the same use case as covered in the first article, whereby an update-item event is…