Prometheus + Grafana Setup To Visualize Your Servers — 2020
Setting up Prometheus, Node Exporter and Grafana to get graphical visualization of server metrics.
Prometheus Introduction
Prometheus is an open source monitoring tool that implements a highly dimensional data model. Prometheus has multiple modes for visualizing data and one of these methods is Grafana integration. Prometheus stores time-series data in memory and on local disk in an efficient custom format.
There are many client libraries available to allow easy instrumentation of services, custom libraries can also be created with ease. Each server is independent for reliability, relying only on local storage. Written in Golang programming language, all binaries are statically linked and easy to deploy.
Prometheus implements the pull policy and not push policy i.e., the prometheus scrapes data from its exporters at definite time intervals rather than them pushing to prometheus, this implementation has its own pros and cons but let’s not get into those details.
Grafana Introduction
Grafana is an open source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. Grafana supports dozens of databases, allowing us to create a single dashboard to visualize all of them.
Grafana also provides alerting, define thresholds visually, and get notified via Slack, PagerDuty and other platforms. Grafana also provides a wide variety of options to view our data, from heat-maps to histograms and graphs to geo-maps. Grafana has a plethora of visualization options to help you understand your data, beautifully. I am using Ubuntu 18.04 and will be showing the entire setup with respect to that itself.
Prometheus Setup
wget https://github.com/prometheus/prometheus/releases/download/v2.21.0/prometheus-2.21.0.linux-amd64.tar.gztar -xzf prometheus-2.21.0.linux-amd64.tar.gzcd prometheus-2.21.0.linux-amd64/./prometheus
The installation is pretty straight-forward, running these commands would result in Prometheus server running in port 9090. The dashboard of prometheus on port 9090 would look something like this:
The metrics scraped from prometheus happens at constant time periods as mentioned before, so they can be viewed in path /metrics
These metrics are used to form complex expressions with various aggregate functions to visualize in the form we desire, this is basically possible with the help of promql. The graph visualization in prometheus is very primitive and doesn’t give much customization so we’ll be using Grafana.
Node Exporter Setup
wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gztar -xzf node_exporter-1.0.1.linux-amd64.tar.gzcd node_exporter-1.0.1.linux-amd64/./node_exporter
The above commands will install the node exporter and runs on port 9100, and the metrics can be scraped from/metrics extension.
I have created 3 virtual machines and installed node exporters in all of them to give better visualization. So after installation, we have to tell prometheus from where to scrape the metrics, which can be done by editing the prometheus.yml file. We just add a new job in the scrape_configs specifying the IP address and port in targets. Once we add the targets in the prometheus.yml file and restart the prometheus server, we can see the new targets and their status in the dashboard as well in the /targets path.
Make sure all the targets are UP, if not check if the 9100 port is exposed for that VM instance. You can also view how much time prometheus took to scrape from each exporter and when it was last scraped.
Grafana Setup
wget https://dl.grafana.com/oss/release/grafana-7.1.5.linux-amd64.tar.gztar -xzf grafana-7.1.5.linux-amd64.tar.gzcd grafana-7.1.5.linux-amd64/./bin/grafana-server
The installation will be done by running the above commands and grafana is hosted on port 3000. The default username and password both are “admin”.
Once we enter the dashboard, we have to add a data source which is Prometheus in our case. We just have to give the prometheus URL and hit the Save & Test button. If we get a success toaster saying data source is working then we are good to go.
We can create our own dashboards and panels with customized queries, but it is a tedious task. So to make our work simple there are already dashboards created by other users. We can use the same and tweak the expressions based on our desire. I am using 1860 and 405, these are the unique ids we give to import the dashboards.
Once the import is complete, we see the graphs based on their expressions and time frame. You can drill down by reducing the time frame to 5 mins.
Grafana also allows us to view a consolidated form of combining multiple exporters placed side-by-side which gives a better picture for comparison purposes.
So in the above images we can see that data is being visualized from all three node exporters.
This sums up the basic setup of Prometheus and Grafana to visualize your node metrics. If you find it helpful, please drop some claps and feel free to share it. This is Vinesh signing-off. Byeee 😋
