Apache Kafka Guide #17 Auto Offset Commit
Hi, this is Paul, and welcome to the #17 part of my Apache Kafka guide. Today we will discuss auto offset commit.
In the Java Consumer API for Kafka, there’s an often-overlooked feature concerning consumer offsets. Regular polling in this API leads to the corresponding commitment of offsets. This process primarily facilitates ‘at least once’ reading patterns under specific conditions.
The key question is: when are these offsets committed? They’re committed whenever a poll is called, provided that a set period, defined by the auto.commit.interval.ms setting, has passed. For instance, with default settings of auto.commit.interval.ms=5000 and enable.auto.commit=true , offsets commit every five seconds.
To maintain an ‘at least once’ setting, it’s crucial to process all messages from Kafka before polling again. Failing to do so means you might commit offsets before processing messages, which deviates from the ‘at least once’ scenario.
In cases where enable.auto.commit is turned off, manual offset commitment is required. This can be done asynchronously or synchronously.
Behind the scenes, offsets are continuously committed. Here’s a quick rundown of the process: With enable.auto.commit=true and auto.commit.interval.ms=5000, the consumer calls .poll(), starting a timer. The Kafka broker returns messages, and subsequent polls fetch more messages. After about six seconds, the system processes data and commits offsets asynchronously since more than five seconds have elapsed since the last poll. This cycle of polling, processing, and committing repeats.
Thank you for reading until the end. Before you go:
- Please consider clapping and following the writer! 👏
- Follow us on Twitter(X), LinkedIn
