Apache Kafka Guide #26 Consumer CLI

Apache Kafka Guide Consumer CLI

Hi, this is Paul, and welcome to the #26 part of my Apache Kafka guide. Today we will discuss how to use Apache Kafka Consumer CLI.

Apache Kafka At the Gates of Mastery: Kafka Core

Introduction

Create a topic with 3 partitions:

kafka-topics --create --bootstrap-server localhost:9092 --topic my-topic --replication-factor 1 --partition 3

We understand that consumers can read data from partitions sequentially. This capability extends to group contexts, which we will explore later. Consequently, we will use a practical exercise to read from a Kafka topic. Our approach will involve consuming from the topic’s end, focusing solely on new messages. Additionally, we will consume from the start of the topic to review all content from its inception. Lastly, we plan to demonstrate options that allow us to view both keys and values in our consumers’ output.

Consuming with CLI

Consuming Latest Messages

To consume the messages we have to use kafka-console-consumer.sh

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic

Consuming all messages from the beginning

To read messages from the beginning, you can use --from-beginning flag

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

Consumer CLI as part of the Group

To read messages as a group member we have to use the flag option:

• --group group-name

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --group group-name

Consuming with MessageFormatter

This section describes our approach, which involves employing a Formatter. Specifically, the Formatter utilizes the default message Formatter to structure the output generated by CLI commands.

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic \
        --formatter kafka.tools.DefaultMessageFormatter \
        --property print.timestamp=true \
        --property print.key=true \
        --property print.value=true \
        --property print.partition=true \
        --from-beginning 

The initial configuration option print.timestamp=trueis set to ensure visibility of the message reception time. Following this, print.key=true it is activated to display the message key, addressing its absence in the default console consumer’s output. Additionally, print.value=true it is enabled to reveal the message’s value. We also include print.partition=true identifying the partition number assigned to the message. Lastly, the --from-beginning setting is applied to facilitate reading messages from the start.

Thank you for reading until the end. Before you go:

Apache Kafka At the Gates of Mastery: Kafka Core

• Please consider clapping and following the writer! 👏
• Follow us on Twitter(X), LinkedIn