Apache Kafka Monitoring with Elasticsearch, Beats, and Kibana

4 min readJun 13, 2020

We can leverage the Elastic stack to monitor Kafka setup. In this article, we will see how to use it in practice.

Introduction

There are several tools available for monitoring Apache Kafka such as Confluent Control Centre, Yahoo Kafka Manager, LinkedIn Burrow, Kafka Tool. More details about each one of them can be found here. Each of them needs a separate installation and configuration. In this article, we are going to see how can we leverage Elasticsearch and Kibana to monitor Apache Kafka.

Apache Kafka

Apache Kafka is a distributed publish/subscribe messaging system with streaming capabilities.

Elasticsearch

Elasticsearch is a distributed search and analytics engine.

Kibana

Kibana is an open-source analytics and visualization platform.

Filebeat and Metricbeat

Filebeat is a lightweight shipper for forwarding and centralizing log data.

Metricbeat is a lightweight shipper that you can install on your servers to periodically collect metrics from the operating system and from services running on the server.

Setup

This guide assumes that Apache Kafka, Elasticsearch, and Kibana are up and running. If not, then please refer to the respective documents in the reference section. However, we will be setting up Filebeat and Metricbeat here.

Since my laptop has Ubuntu, I’ll be using APT repository to install Filebeat and Metricbeat.

NOTE: The beats should be installed on Kafka servers and Elasticsearch cluster and Kibana should be accessible from there.

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - 
$ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
$ sudo apt-get update sudo apt-get install filebeat metricbeat 
$ sudo systemctl enable filebeat.service 
$ sudo systemctl enable metricbeat.service

Now, we need to change the Elasticsearch and Kibana configuration in /etc/filebeat/filebeat.yml and /etc/metricbeat/metricbeat.yml

#====================== Kibana =====================================# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:# Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5602"# Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:#--------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]# Protocol - either `http` (default) or `https`.
  #protocol: "https"# Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

Once this is done, we’ll have to enable the kafka and system modules for both the beats:

$ filebeat modules enable kafka system
$ metricbeat modules enable kafka system

Once enabled, we need to set up both the beats:

$ filebeat setup -e --modules kafka,system
$ metricbeat setup -e kafka,system

By default, filebeat looks for Kafka under /opt . If you have installed Kafka at a different location, you need to specify it in /etc/filebeat/modules.d/kafka.yml as shown below:

# Module: kafka
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.7/filebeat-module-kafka.html- module: kafka
  # All logs
  log:
    enabled: true# Set custom paths for Kafka. If left empty,
    # Filebeat will look under /opt.
    var.kafka_home: /home/imteyaza/Downloads/kafka_2.12-2.1.0# Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
        - /home/imteyaza/Downloads/kafka_2.12-2.1.0/logs/*

Once both the beats are configured we need to start them:

$ sudo systemctl start filebeat.service
$ sudo systemctl start metricbeat.service

Now open Dashboard tab in Kibana and type kafka in the search box, you should see two results as shown below:

Open the [Metricbeat Kafka] Overview ECS. It shows the current state of the cluster. We can select a particular topic from the dropdown.

NOTE: You won’t be able to see any data initially, if you are not actively putting data in Kafka.

Now, let’s open the [Filebeat Kafka] Overview ECS . It shows the log details of Kafka. The Stacktraces shows the exceptions that occurred in the selected time frame.

Kafka Fileabeat Dashboard — Filebeat Dashboard

We can use these dashboards for monitoring Kafka in real-time.

In this article, we saw how to use Elasticsearch, beats(filebeat and metricbeat ) and Kibana for monitoring Apache Kafka. Hope it helps and happy learning 😊.

References

Putting Data in Kafka

Create a Kafka topic

imteyaza@imti:~/Downloads/kafka_2.12-2.1.0$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafka-monCreated topic "kafka-mon".

Send data

imteyaza@imti:~/Downloads/kafka_2.12-2.1.0$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kafka-mon>this is a message
>This is another message
>Sending another text message     
>Another test message
>

Consuming Data

imteyaza@imti:~/Downloads/kafka_2.12-2.1.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka-mon --from-beginningthis is a message
This is another message
Sending another text message
Another test message

Apache Kafka Monitoring with Elasticsearch, Beats, and Kibana

Introduction

Setup

References

Putting Data in Kafka

Written by Imteyaz Ahmad