Accessing Kafka Broker

Access is important

This is a fifth article in our Running Kafka in Kubernetes publication. Refer to https://medium.com/kafka-in-kubernetes/deploying-kafka-broker-cluster-5ba2790fdb5b to see the previous article.

Motivation

Without testing, we will never be sure if our Kafka setup is working. In this article, you will see how we perform the Kafka produce and consume from within the cluster and also from external.

Access from Internal

This section is about accessing the Kafka Topics from a Pod inside the same K8S cluster.

To access the Kafka broker from internal, we created a Kafka-Cli docker image.

The Kafka CLI docker will be used to run the Kafka CLI to produce and consume topic messages from within the cluster. Please refer to https://github.com/fernandocyder/k8s-practice/tree/master/04.kafka-expose-service for details of the Dockerfile and other related build files.

This is the docker URL after built successfully, https://hub.docker.com/repository/docker/fernandokarnagi/kafka-cli

Deploy that into the K8S cluster, kafka namespace. The file is here https://raw.githubusercontent.com/fernandocyder/k8s-practice/master/04.kafka-expose-service/03.kafka-cli.yaml

Let’s start the testing.

Access the shell of the kakfa-cli POD

k exec -it pod/kafka-cli -- /bin/bash

From within the kafka-cli POD, run the following commands

kafka-topics.sh --zookeeper zookeeper-0.zookeeper:2181 --list
kafka-topics.sh --zookeeper zookeeper-1.zookeeper:2181 --list
kafka-topics.sh --zookeeper zookeeper-2.zookeeper:2181 --list
kafka-topics.sh --zookeeper zookeeper-service.kafka.svc:2181 --list

The result is OK, no error, and nothing comes out, which is true, since there is no Topic at the moment.

Now create a new Topic.

From within the kafka-cli POD, run the following commands:

kafka-topics.sh --create --zookeeper zookeeper-service.kafka.svc:2181 --replication-factor 1 --partitions 1 --topic testtopic

The verify using all the above 4 commands

The above screenshot shows that the ‘testtopic’ is created successfully can be seen from all zookeeper nodes (containers).

Now, test produce and consume

From within the kafka-cli POD, run the following commands

kafka-console-consumer.sh --bootstrap-server kafka-0.kafka:9092 --topic testtopic --from-beginning

From within the kafka-cli POD, run the following commands

kafka-console-producer.sh --bootstrap-server kafka-0.kafka:9092 --topic testtopic

This is the result of the produce

This is the result of the consume

Access from External

This section is about accessing the Kafka Topics from my own Mac.

Let’s start the consumer

./kafka-console-consumer.sh --bootstrap-server broker.mydomain.com:19092 --topic testtopic --from-beginning

Then start the producer

./kafka-console-producer.sh --bootstrap-server broker.mydomain.com:19093 --topic testtopic

Once the producer is launched, you can start typing messages and you can then see the messages are seen in consumer

Below screenshot shows the producer

Below screenshot shows the consumer

Conclusion

You have seen how the access to Kafka Broker is fine from both internal and external.