Monitoring Kafka and data within it , in real time

So we all know that Kafka is a messaging system . Well this definition looks fancy but actually it is a very powerful high throughput and easily configurable streaming solution . It can be made to operate in real time and in near real time , depending on your use case . For simplicity and making people to easily understand this use case i would use the term “data” in place of “message”.

Data in Kafka flows in the forms of bytes . That’s the very reason you need to mention the serializer you opt in your client or custom connector while writing the code for Producer or Producer connector . Based on that the data which is being send ,is serialized into bytes and transferred onto the network . That’s the very reason why Kafka is so fast . For Kafka data is nothing but just a stream of bytes which needs to be transported over the network . Data type and data structure doesn’t matter to Kafka as internally Kafka is unaware about the data . Is that even true ? . Streaming enthusiasts might ask that . Well i agree to disagree. Let’s find out how .

In Kafka actual data flows in the form of offsets . Each data records is mapped to an offset .These offsets are used to ensure ordering of data within a partition and also acts as a marker on the data bytes flowing in the system.So isn’t it convenient to tag data to offset . So if we can find a way to track the offsets and make sense out of that information then we can break this jinx of deciphering Kafka operations in real time .Let’s cover this journey together . We will need 3 things for this task .

i)A working Kafka cluster

ii)A TSD Database(Time series Database)

iii)BI tool which can be easily coupled with TSD DB .

Now one question might be coming to your mind . Why TSD DB only why not any relational DB . Well the answer is that TSD databases are the most optimized one for storing metrics level monitoring information and millisecond query configuration . Ok So enough theory . Let’s dive deep into this demonstration .

I used below ecosystem tools for this demonstration :

i)AWS platform (cloud formation stack to spawn my ec2 instances with EBS storage to host Kafka cluster . Kept it a 3 broker configuration with 1 zookeeper for simplicity.)

ii)Prometheus :- Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database built using a HTTP pull model, with flexible queries and real-time alerting.

iii)Grafana: It is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. It is very tightly coupled with Prometheus .

Kafka Cluster architecture looks like this :

Will not delve into creating kafka cluster . Will take it from the point where in we have a running Kafka cluster since a cluster can be assumed to running on an on premise or any cloud server.

So once we have a running Kafka cluster we need to install Prometheus on the existing server box . Take these following steps:

i)cd /home/ec2-user

ii)mkdir prometheus

iii)cd prometheus

iv)wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar

v)wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml

This step is performed to pull Kafka metrics in a particular format in Prometheus .

Next and the most important step is to add JMX java prometheus agent in Kafka service file .

Updated Kafka service file with JMX detailed added .

Then restart kafka demon and restart kafka

i)sudo systemctl daemon-reload

ii)sudo systemctl restart kafka.

then check kafka status again . It should be active and running .

Doing a curl to check Kafka restart status

Now to verify if you have started receiving kafka metrics in Prometheus we need to do curl . Prometheus is pointing to port 8080.

Prometheus check at port 8080

Now go the folder where Prometheus is installed and check Prometheus status . You should start seeing following logs .

Now open Prometheus on web browser and following application should load .

Prometheus application

Now we can see all sort of metrics of Kafka in Prometheus

Kafka metrics in Prometheus

Now we need to add Prometheus as a system property as this will add it as an automated involved component in your Kafka installation and you need not to trigger it every time manually .

Now we need to follow these steps right from the point where we added Prometheus on Kafka.service file for broker 2 and broker 3 machines .

Now wait, where are the zookeeper log information . That is not coming into Prometheus . So now let’s work towards that . Let’s add Prometheus JMX java agent in zookeeper.service file

Now restart Zookeeper .

Post this we should start seeing zookeeper metrics as well in Prometheus

So we are good with Prometheus now . Let’s start with Grafana . Follow these steps

i)wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.2.1.linux-amd64.tar.gz

ii)tar -zxvf grafana-*.gz

iii)rm grafana-*.gz

iv)mv grafana-*/ grafana

v)cd grafana

In Grafana set up we need to edit default.ini file and make anonymous enabled = true

Now start Grafana service

Also keep in mind that you need to enable following ports in your administration machine or centralized nodes to allow incoming traffic .

Now load Grafana Dashboard

Let’s start importing data from Prometheus into Grafana now

Now we need to add Prometheus as a Data source to Grafana

Once this is done then we are reaching to the end of our Demo and you should be able to see these Kafka metrics in real time .

We can configure further what metrics we need to import in Grafana Dashboard . For that there will be a separate post so stay tuned.