Kafka Consumer Auto Offset Reset

The auto offset reset consumer configuration defines how a consumer should behave when consuming from a topic partition when there is no initial offset. This is most typically of interest when a new consumer group has been defined and is listening to a topic for the first time. This configuration will tell the consumers in the group whether to read from the beginning or end of the partition.

Consuming Messages

Every Kafka consumer belongs to a consumer group, grouped together by the consumer’s group.id configuration setting. A consumer group will contain one or more consumers. The consumers within the consumer group will be assigned to topic partitions in order to consume their messages. Each partition will have only one consumer assigned to it, although a consumer may be assigned to multiple partitions within any one topic, and similarly assigned to partitions in all the topics it is subscribed to.

When a new consumer group is first created and its consumers are assigned to topic partitions, they must decide from which point to start polling messages. Unless the consumer has been told to poll from a specific offset (a less common scenario), there are two main options. First a consumer may read messages from the beginning of the partition, processing every message present on the partition log. The second option is to only…