How avoid idempotency problem from Apache Kafka Consumers with in-memory data structure store “Redis” ?

Consuming messages from Kafka with multi-thread application may cause the idempotency problem.

Idempotency problem means; “applying the same operation multiple times has the same effect as applying it once”.

https://serverlessland.com/assets/visuals/eda/understanding-idempotency.png

So how can we do this? Because consuming a message multiple times from Kafka will cause unexpected multiple same service calls or doing same operations.

For example, there is a Kafka topic with “SEND-MAIL-TO-SUBSCRIBER” and this topic has a message. When consuming this message multiple times from Kafka will cause the send many same mails to suscribers.

To avoid from this problem, use Redis as a in memory-cache;

First of all,

Define an unique-identifier for your Kafka message to store this message to Redis with this identifier.

try {
    this.redisRepository.put(mailId,mail);
} catch (Exception e) {
}

And then check this identifier in Redis before consuming a new message.

If the identifier is exist that stored before in Kafka , skip this message and continue but if it is not stored define another identifier for new message and put it in Kafka.

try {
   this.redisRepository.find(mailId);
    if (isPresent()) {
        retrievedMessage()
    }
} catch (Exception e) {
}

Finally;

This approach will avoid idempotency problem from your applications.