Journey of Apache Kafka & Zookeeper Administrator ( Part 5 )
June 2019 ( continued and last part of June )
Today, We need to talk about Another Operations aspects, Once you have Apache Kafka installed, how to manage topics and other basics things for Apache Kafka, that’s where I got to know about Yahoo Kafka Manager ( aka CMAK ).
I want to thanks #yahoo for a really great tool.
Let’s talk about why this tool is so great.
It supports multiple clusters and can do so many operations very easily like Topic Creation, Topic Config Update, Broker Overview, Live Broker Stats ( Number of Messages / Number Partitions / Active Partitions ) and Can do Super Important Task aka Partition Reassignment with just a couple of clicks.
This is very easy to install and even easier to Onboard Other Administrator who is just starting Kafka.
There was Another good tool, I tried called Kafdrop. It was really great because you can actually see messages in real-time from any topic if you need to see them which is still lacking in CMAK. Anyways I decided to move forward with CMAK instead of Kafdrop which bit mature and have a community that can support it in long term.
How to Install CMAK Manually?
I recommend that you build the tool yourself, download the latest release: github.com/yahoo/CMAK/releases.
Step 1:
Install Open Java 11 and build using simple command or follow steps from the GitHub repository.
./sbt clean distThe above command will generate a zip file with cmak.xx.zip which contains all the library + binaries to start CMAK.
Step 2:
Copy the Above zip to cmak server and unzip it on your given partition/location.
update application. conf to include
- basic authentication details like username and password.
cmak.zkhosts="10.0.0.1:2181,10.0.0.2:2181,10.0.0.3:2181"
Step 3:
Create a SystemD file so you can start it as a Service.
[Unit]
Description=Apache Kafka Manager ( CMAK )
After=syslog.target network.target[Service]
Type=simple
User={{ kafkaUser }}
Group={{ kafkaGroup }}
ExecStart={{ kafkaManagerInstallDir }}/kafka-manager/bin/cmak[Install]
WantedBy=multi-user.target
Step 4:
Start the Service with the below command
service kafka-manager startHow to Install CMAK with Ansible?
116davinder/kafka-cluster-ansible/clusterKafkaManager:
This playbook will install Kafka Manager aka CMAK as mentioned in the above steps. Now we are using ansible almost everywhere which perfect example of Configuration Management.
Again I am following the same folder structure which is to point to the latest version using symbolic link aka soft link.
davinderpal@Davinder:~$ tree -L 1 kafka-manager/
kafka-manager/
├── cmak-3.0.0.4
├── cmak-3.0.0.5
└── kafka-manager -> cmak-3.0.0.53 directories, 0 files
Note*
I am using Basic Authentication because Only DevOps Team is required access to this UI and Basic Authentication is far easy than Implementing LDAP but you can use LDAP as well if required.
Next Steps
Once install is done via Ansible or Manually, We need to Add clusters to CMAK so it can start Monitoring / Administrating them.
Login to CMAK GUI on Port 9000 with your Credentials.
Click to Clusters and then click Add Clusters, just fill in details about Cluster
- cluster name
- zookeeper nodes for given Kafka cluster
- other parameters like CMAK threads + JMX settings + JMX authentication if implemented
Once you submit the form, it will show your cluster in clusters link as shown below.
If anything else is required, please read CMAK documentation https://github.com/yahoo/CMAK.
Above is what my requirements were at that time.
The journey will continue on Next Topic Apache Kafka Mirror Maker V1.
