Apache Kafka Guide #25 Producer CLI

Hi, this is Paul, and welcome to the #25 part of my Apache Kafka guide. Today we will discuss how to use Apache Kafka Producer CLI.

Apache Kafka At the Gates of Mastery: Kafka Core

Introduction

First, we’ll proceed without transmitting keys, resulting in a null key, and the data will be evenly distributed among all partitions. Alternatively, we can send with keys, ensuring that identical keys are consistently directed to the same partition.

Just a quick reminder of what data flow looks like:

Create topic:

kafka-topics --create --bootstrap-server localhost:9092 --topic my-topic --replication-factor 1 --partition 1

Producing Messages without Keys

To produce the data using kafka-console-producer.sh:

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-topic

Alright, let’s proceed and execute this command. Upon initiating your console producer, you will notice a chevron > on the left side, indicating that it’s prepared for production. Therefore, I’ll simply input “Hello world.”

Every time you hit the enter key, it effectively sends a message into Kafka, marking the creation of a new Chevron. To cease transmitting messages into Kafka, simply use the Control+C command to exit the console producer.

Subsequently, we have the opportunity to improve our Producer.

By defining specific properties, we aim to augment its functionality, altering aspects like batching, among others. This is achieved through the use of the producer property argument.

For instance, in the scenario presented, I employ properties for production, including setting the property acks to all. This is to ensure that every message receives acknowledgments from all brokers.

Just add to our previous command --producer-property acks=all

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-topic --producer-property acks=all

Therefore, you will not observe any changes on the producer side. However, on the backend, we have a requirement acks=all.

Producing Messages with Keys

By default, utilizing the console producer to dispatch messages results in the transmission of a null key. However, the capability exists to send messages with specified keys. This practice ensures that messages sharing the same key are directed to the identical partition, a phenomenon we plan to explore further.

Upon examination of key-enabled message production, the command to execute this action is replicated here, alongside a review of its parameters. The target remains my-topic, a pre-existing entity within our scope, and the command includes a directive to parse the key, signified by parse.key=true. Additionally, it specifies a key.separator, denoted by a colon. Consequently, the message structure positions the key to the left of the colon and the value to its right.

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-topic --property parse.key=true --property key.separator=:
> key:value

Currently, illustrating the principle that identical keys are assigned to the same partition is impractical due to the presence of only one partition, which means all messages converge on this singular partition. However, this behavior will be substantiated in subsequent demonstrations.

Employing a producer configured with a key separator necessitates caution; omitting the colon in a message triggers an exception, indicating the absence of a discernible key separator.

Topic Auto Creation

When attempting to send messages to a topic that doesn’t exist, you’ll encounter different behaviors depending on the setup of your clusters. For instance, in our scenario, the system appears to operate correctly at first glance, as indicated by a chevron symbol. However, when attempting to send a “Hello World” message, an error or timeout occurs. This issue arises because our system is configured to prevent message production to non-existent topics, as evidenced by the error message indicating the absence of the topic in the metadata, confirming it was never established.

To further understand this, conducting a Kafka topics command to list available topics reveals that only the initial topic exists. This contrasts with the behavior observed when executing the same command on a Kafka instance on localhost with default settings. In such cases, attempting to send a “Hello World” message results in warnings about the leader not being available due to the lack of a leader for the automatically created topic. Initially, these errors may appear, but after a few attempts, the topic is created with a designated leader, allowing the “Hello World” message to be successfully sent. This can be verified by listing the topics, where the newly auto-created topic will be visible, typically configured with a single partition as per the default Kafka cluster settings.

To modify this behavior, adjustments can be made in the server.properties file used to start the Kafka server. For instance, setting the number of partitions to three means that any new topic created automatically will have three partitions by default. This adjustment simplifies the initial Kafka setup process. However, it’s important to note that the best practice, widely adopted across various clusters, is to disable automatic topic creation. Users are strongly advised to pre-define their topics, a policy we have implemented by disabling the auto-creation of topics.

Thank you for reading until the end. Before you go:

Apache Kafka At the Gates of Mastery: Kafka Core

• Please consider clapping and following the writer! 👏
• Follow us on Twitter(X), LinkedIn