When I was preparing some demos about Amazon Managed Streaming for Kafka (Amazon MSK) I realised that one of the CLI-tools that I like to use was not available on my EC2 instance: the useful open-source kafkacat tool.
There seems to be no binary available for Amazon Linux via yum install, but you can quickly build it yourself. Doing so requires UNIX make, as well as a C and C++ compiler (so the whole task mentally throws you back to the time before the “write once, run anywhere” became a thing).
kafkacat Installation on Amazon Linux2
To build kafkacat yourself, just run the following commands on the EC2 instance that you want to use as a client (where you want to install kafkacat):
sudo yum update -y
sudo yum install git -y# make sure you have the necessary dev tools avail,
# or run the ff command:
sudo yum group install “Development Tools” -ygit clone https://github.com/edenhill/kafkacat.git
You might not need or get away with less then the “Development Tools” group install, let me know if you find out.
Then try out your newly installed kafkacat tool, e.g. by using one of the following commands:
./kafkacat -h./kafkacat -L -b YOUR.kafkabroker.com:9092
You can use the AWS MSK console to retrieve the bootstrap servers easily:
For an MSK cluster set up with plain text, you can query for the brokers as shown below:
./kafkacat -L -b b-1.kafkathree.k4hgaf.c3.kafka.eu-central-1.amazonaws.com:9092,b-3.kafkathree.k4hgaf.c3.kafka.eu-central-1.amazonaws.com:9092,b-6.kafkathree.k4hgaf.c3.kafka.eu-central-1.amazonaws.com:9092
In this example I have 6 broker running across 3 availability zones:
There are many other examples that you might find useful when running Apache Kafka especially on AWS. I will continue this posting with some more examples… stay tuned!
This article about calling MSK via REST might be interesting.