How to implement retry logic with Spring Kafka

In this article, I am going to explain our approach for implementation of retry logic with Spring Kafka.

Imagine an application that consumes messages from Kafka and updates its data according to the information contained in the message. What would you do if your system can’t successfully process the message on the first attempt? If your application doesn’t have any retry logic you lost the message and failed to update your data in case of an error.

Let’s see our retry logic design.

As you can see above, a sample listener consuming messages from “sample-topic” with a configured container factory and consumer group id.

If an exception occurs message is sent to the “sample-topic_sample-consumer-group_RETRY” topic with the message key. Also, we added exception to message to determine the root cause of the error.

Another point I would like to mention here, we named retry topic with “topicName + _consumerGroupId_ + RETRY”. The reason we named it like this is that there may be other applications consuming the same topics with a different consumer group, and we should be able to tell which one fails and how many error messages there are for each one, then run appropriate logic for each.

Then, the application consumes retry topic with configured retry container factory as above and run its logic.

In this way, application doesn’t run retry logic on the main topic (“sample-topic”). So, the consumer of the main topic will never be obstructed.

Let’s take a look consumer configurations

We configured two different kafka listener container factory to use in the consuming main topic and retry topic.

The kafkaListenerContainerFactory configuration used by main topic is defined as above.

The retry configuration used by retry topic is defined as above. This configuration uses Spring RetryTemplate (don’t forget to add @EnableRetry annotation to main class). Thus, you can set the number of attempts and backoff period. According to the retry configuration, retry will be executed 3 times in 3000ms interval.

If all retries are exhausted you can run your logic inside recovery callback. We send messages to another topic which named like this “topicName + _consumerGroupId_ + ERROR” pattern.

You should decide what you can do with error messages in the error topic. You can define an alert or send messages to your team’s slack channel.

There is also another way to run retry logic with new versions of Spring Kafka.

factory.setErrorHandler(new SeekToCurrentErrorHandler(new DeadLetterPublishingRecoverer(kafkaTemplate), 3));

With the config above, kafka consumer retries 3 times in case of error and publishes the message to “topicName + .DLT” topic.

We didn’t prefer this way due to this configuration executes retry logic on the main topic and you are not free to giving a name for error topic. Also it blocks the main consumer while its waiting for the retry, which causes your system to slow down significantly in case on erroneous message.

If you’d like to work on projects such as this, or develop/contribute on our open source projects you know what to do just come to Trendyol :)