How avoid idempotency problem from Apache Kafka Consumers with in-memory data structure store “Redis” ?
Consuming messages from Kafka with multi-thread application may cause the idempotency problem.
Idempotency problem means; “applying the same operation multiple times has the same effect as applying it once”.
So how can we do this? Because consuming a message multiple times from Kafka will cause unexpected multiple same service calls or doing same operations.
For example, there is a Kafka topic with “SEND-MAIL-TO-SUBSCRIBER” and this topic has a message. When consuming this message multiple times from Kafka will cause the send many same mails to suscribers.
To avoid from this problem, use Redis as a in memory-cache;
First of all,
Define an unique-identifier for your Kafka message to store this message to Redis with this identifier.
try {
this.redisRepository.put(mailId,mail);
} catch (Exception e) {
}
And then check this identifier in Redis before consuming a new message.
If the identifier is exist that stored before in Kafka , skip this message and continue but if it is not stored define another identifier for new message and put it in Kafka.
try {
this.redisRepository.find(mailId);
if (isPresent()) {
retrievedMessage()
}
} catch (Exception e) {
}
Finally;
This approach will avoid idempotency problem from your applications.