TECH BASICS

Monitoring Kubernetes Cluster using Elastic Stack( Elasticsearch-Kibana-Metricbeat)

Monitoring With Elastic

Arun Kumar Singh
Apr 5 · 7 min read

Objective:

In this medium article, we are going to deploy the Elastic Stack (Elasticsearch-Kibana-Metricbeat) for monitoring the Kubernetes Cluster. Metricbeat will collect metrics from Kubernetes Cluster and it will forward the data to Elasticsearch for analytics. Kibana will allow us to visualize data in a dashboard format. And the best part is that We will deploy this whole stack on Kubernetes Itself. This article consists deployment of all component in the simplest way so do not worry/panic if you see the password in plain YAML 😬 😐

What is Elastic Stack?

Elastic Stack is the collection of three open-source products Elasticsearch, Kibana, Logstash and Beats. A few years back Elastic added the Beats component as well. Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack. It centrally stores your data for lightning-fast search. Kibana enables you to interactively explore, visualize, and share insights into your data and manage and monitor the stack. Logstash and Beatsfacilitate collecting, aggregating, and enriching your data and storing it in Elasticsearch. Beats are single-purpose data shippers and of different types.

Beats from Elastic (https://www.elastic.co/beats/)

Metricbeat?

Metricbeat is the heart of this whole monitoring stack which we are planning to deploy. Metricbeat helps you monitor your servers and the services they host by collecting metrics from the operating system and services. In real-world scenarios, you can deploy Metricbeat on all your Linux, Windows, and Mac hosts, k8s clusters connect it to Elasticsearch. You will get system-level CPU usage, memory, file system, disk IO, and network IO statistics, as well as top-like statistics for every process running on your systems. You can transform or enrich your metrics then use Logstash before sending it to Elasticsearch.

How Metricbeat Monitors?

Metricbeat comes with internal modules that collect metrics from services like Apache, MongoDB, Prometheus, containers, etc. To make it work you need to just enable the modules. These modules are part of your configuration yml.

Metricbeat deployment and monitoring mainly involve the following steps -

  • install Metricbeat on each node/system which you want to monitor
  • specify the metrics you want to collect
  • send the metrics to Elasticsearch or you can send to logstash for data enrichment and then to Elasticsearch
  • visualize the metrics data in Kibana using OOTB dashboards or build your own
METRICBEAT SENDING DATA TO ELASTIC

What do we have?

I have created a Kubernetes Cluster using Kubeadm. This cluster will be used for all deployments.

How we will do it?

We will deploy Elasticsearch-Kibana Stack on Kubernetes using ECK Operator. Using ECK Operator, Elasticsearch and Kibana can be deployed in Seconds. Metricbeat deployment on Kubernetes can be done in multiple ways. We have the following options -

  1. Using Standard YAML
  2. Using Helm Charts
  3. Using ECK Operator

I will use Helm Charts for this time to do the deployment.

What will be the steps?

  1. Deploy ECK Operator
  2. Using ECK Operator deploy Elasticsearch-Kibana
  3. Deploy Metricbeat using Helm Chart
  4. Load Metricbeat Dashboards
  5. See the Magic

Step 1: ECK Operator Deployment

ECK Operator deployment is pretty straightforward. Simple deployment can be done using YAML file.

$ kubectl apply -f https://download.elastic.co/downloads/eck/1.5.0/all-in-one.yaml

Once the Operator is in place following CRDs are available to use-

agents.agent.k8s.elastic.co
apmservers.apm.k8s.elastic.co
beats.beat.k8s.elastic.co
elasticsearches.elasticsearch.k8s.elastic.co
enterprisesearches.enterprisesearch.k8s.elastic.co
kibanas.kibana.k8s.elastic.co
ECK Operator CRDs

Step 2: Elasticsearch-Kibana Deployment

Now ECK Operator is in place, so we can move ahead to deploy Elasticsearch and Kibana components. I will deploy a very basic cluster with minimum requirements. Attached YAML is for ref:

arun@controller:~/monitoring$ kubectl apply -f mon_elm.yaml (github.com)
elasticsearch.elasticsearch.k8s.elastic.co/elsmonitor created
kibana.kibana.k8s.elastic.co/kibmonitor created
##### wait for few seconds and check the cluster status ###arun@controller:~/monitoring$ kubectl get pods
NAME READY STATUS RESTARTS AGE
elsmonitor-es-default-0 1/1 Running 0 73s
kibmonitor-kb-d4b6f847-7nvqg 1/1 Running 0 72s
arun@controller:~/monitoring$ kubectl get elasticsearch
NAME HEALTH NODES VERSION PHASE AGE
elsmonitor green 1 7.12.0 Ready 9m39s
arun@controller:~/monitoring$ kubectl get kibana
NAME HEALTH NODES VERSION AGE
kibmonitor green 1 7.12.0 10m

When you deploy the cluster using ECK, it generates the password for the elastic superuser and keeps it a secret.

PASSWORD=$(kubectl get secret elsmonitor-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')

Use this password to login cluster. For quick verification use the svc port-forward method to access the service on localhost.

arun@controller:~/monitoring$ kubectl port-forward svc/kibmonitor-kb-http 30135:5601
Forwarding from 127.0.0.1:30135 -> 5601
Forwarding from [::1]:30135 -> 5601
## Now you can access the service directly. > http://localhost:30135
Kibana Login Screen

Step 3: Deploy Metricbeat Using Helm Chart

Elastic provides the Helm Chart for Metricbeat. We will use this Chart for the time.

$ git clone https://github.com/elastic/helm-charts.git
$ cd helm-charts/metricbeat
$ helm install mbeat . -f custom_Values.yaml
Helm Install

I have prepared one custom values.yaml as per our requirement. This YAML file contains information about Elasticsearch and Kibana endpoints.

Why Kibana Endpoints? Because I want to deploy OOTB available Metricbeat Kubernetes dashboards as well.

When you deploy this chart, the following main component gets deployed at once.

  1. Metricbeat as a DaemonSet to ensure that there’s a running instance on each node of the cluster. These instances are used to retrieve most metrics from the host, such as system metrics, Docker stats, and metrics from all the services running on top of Kubernetes.
  2. Single Metricbeat instance using Deployment. This instance is used to retrieve metrics that are unique for the whole cluster, such as Kubernetes events or kube-state-metrics.
  3. kube-state-metrics to generate metrics about the state of the Kubernetes Cluster Objects.
  4. A service for kube-state-metrics

Please note you don't have to run DaemonSet and deployment of Metricbeat at the same time. In case of Large K8s cluster run a dedicated, standalone Metricbeat instance using a Deployment in addition to the DaemonSet.

Once deployment is completed, check the Index Management section of Elasticsearch to view Metricbeat Indexes in Kibana.

Metricbeat comes packaged with various pre-built Kibana dashboards that you can use to visualize metrics about your Kubernetes environment. You can run this command from POD or install metricbeat on any node which can reach Kibana.

./metricbeat setup --dashboards

You can check loaded Dashboards in the dashboard section of Kibana.

All Dashboard in Kibana

Search Kubernetes Dashboard and click on Overview ECS one. Lets find if we can see out cluster data.

Voila! Look at the beautiful dashboard below.

Metricbeat Kubernetes Dashboard

That’s it for this post. In the next post, I will show how to deploy Metricbeat using ECK.

Stay Safe and Keep learning. 🌎

PS: If this article has been useful to you, please leave few claps! (you are allowed up to 50)

Nerd For Tech

From Confusion to Clarification

Arun Kumar Singh

Written by

In quest of understanding How Systems Work !

Nerd For Tech

NFT is an Educational Media House. Our mission is to bring the invaluable knowledge and experiences of experts from all over the world to the novice. To stay up to date on other topics, follow us on LinkedIn. https://www.linkedin.com/company/nerdfortech

Arun Kumar Singh

Written by

In quest of understanding How Systems Work !

Nerd For Tech

NFT is an Educational Media House. Our mission is to bring the invaluable knowledge and experiences of experts from all over the world to the novice. To stay up to date on other topics, follow us on LinkedIn. https://www.linkedin.com/company/nerdfortech

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store