Monitoring Elasticsearch with Prometheus and Grafana

Happy devOps

(λx.x)eranga

Published in

Effectz.AI

4 min readApr 30, 2019

Prometheus and Grafana

Prometheus and Grafana becoming most common monitoring platform in microservices based devops infrastructure. Prometheus is a powerful time series metrics collection and alerting system. Grafana is a visualization tools which can be used with Prometheus.

Metrics collection of Prometheus follows the pull model. That means, Prometheus is responsible for getting metrics from the services that it monitors. This process introduced as scraping. Prometheus server scrapes the defined service endpoints, collect the metrics and store in local database. Then Grafana can visualize them in various graphs and charts. To query the metrics from Prometheus, it provides a new query language called PromQL. We can define the PromQL queries with Grafana to get data from Prometheus.

Expose metrics

In order to Prometheus to scrape the metrics, each service need to expose their metrics(with label and value) via HTTP endpoint /metrics. For an example if I want to monitor a microservice with Prometheus I can collect the metrics from the service(ex hit count, failure count etc) and expose them with HTTP endpoint. To do that I need to create HTTP server with /metrics endpoint inside the microservice. Prometheus comes with different client libraries to do that.

Exporters

Promethus provides a way to monitory third party applications and services with Exporters. Exporters act as side-car to third party application/services. They collect data from third party applications/services and expose them with HTTP endpoint which Prometheus can scrape. There are various exporters available with Prometheus. The available exporters can be find from here. The most common exporter is node exporter, which can be installed on every server to read system level metrics such as cpu, memory, file system etc.

1. Elasticsearch Exporter

To monitor the metrics of elasticsearch there is an exporter available elasticsearch_exporter. It can monitor elasticsearch cluster health statuses, index statuses, sharing etc. You can find more informations about elasticsearch_exporter in here. In this post I’m gonna show how to visualize elasticsearch metrics with Prometheus and Grafana by using elasticsearch_exporter. All the deployments which relates to this post available in this repo. Please clone it and follow the below steps.

2. Run Elasticsearch Exporter

First I need to run elasticsearch_exporter. Following is the docker-compose.yml for that. elasticsearch_exporter collects metrics from elasticsearch and expose with 9108 port.

3. Run Prometheus

Next I need to run the Prometheus. Following is the docker-compose.yml for that. It defines the volume mapping for prometheus configuration file.

Prometheus configuration file promethus.yml resides on my current working directory. Following is the content of the config file. It defines two scrapers, one to collect metrics of docker and another one to collect statistics of elasticsearch.

Now Prometheus scraping the metrics(in each 5 second) which expose from elasticsearch_exporter. The collecting metrics will be stored in prometheus local database.

4. Run Grafana

Now I can run Grafana and visualize the elasticsearch metrics available on Prometheus. Following is the docker-compose.yml to run Grafana.

5. Grafana data source

To visualize data on Grafana, first need to add data source. In this case Prometheus is the data source. Following are the steps to add the data source in Grafana.

6. Grafana dashboard

After adding prometheus data source I can create a dashboard and visualize the data. Grafana provides a feature to import the dashboards as JSON. There are various dashboard which built by open source community. We can simply take them and use them as our dashboards. You can find available dashboards in here. I have already taken the dashboard which built by elasticsearch_exporter and put in it here. Following is the way to import and use that dashboard.