Exactly-once semantic Kafka with Nifi enabled finally !
Step-by-step guide to implement real-time analysis with Exactly-once delivery enabled
A Few days ago, I had to work on Real-time analysis project on stock market data using Apache Nifi and Apache Kafka. Any one who has used Apache Kafka before would know that “Exactly-Once delivery” needs additional implementation, the challenge I was facing is how to enable the exactly-once in Kafka inside of Nifi. After some research, I found a video that introduce new features added to Apache Nifi in version 1.15 that will make that possible.
Since I didn’t find enough articles that point to this problematic, I have decided to share a demo on how to use “Exactly-Once semantic ” in Kafka with Nifi.
Prerequisites
- Docker, Docker Compose
- Basic knowledge in Apache Nifi, Apache Nifi Registry and Apache Kafka
What is Exactly-Once semantic in Kafka ?
In Apache Kafka, “Exactly-Once semantics” refers to a processing guarantee that ensures each message is delivered and processed exactly once, without any duplicates or message loss.
