Kafka Consumer Non-Blocking Retry Pattern
When an event consumed from Kafka needs to be retried, the retry can either be blocking or non-blocking. Blocking retry means that subsequent events on the partition will not be processed until the retrying event has completed successfully or otherwise. Non-blocking retry means that those later events can be consumed and processed while the original event is being retried. The trade-off for achieving non-blocking retry is that the order of events being processed can no longer be guaranteed.
This is the second of a two part series on non-blocking retry. In the first part non-blocking retry using Spring Kafka was detailed. Spring Kafka provides excellent support with minimal coding required to implement a non-blocking retry pattern. However, for applications not written using Spring, or indeed not using Apache Kafka, then this article describes the design for a generic non-blocking retry pattern that could be applied.
The source code for the accompanying Spring Boot demo application is available here.
For a general overview on consumer retry, and for use cases that require blocking retry, see the article Kafka Consumer Retry.
Non-Blocking Retry Pattern
This article covers the same use case as covered in the first article, whereby an update-item event is…