Multi-Cloud Monitoring at Simplilearn — Part II

Photo by Campaign Creators on Unsplash

In Part I of the Multi-cloud Monitoring blog, we examined the installation of Grafana with Multi-cloud data source setup. In this blog, we deal with the installation of Prometheus and create dashboards using two of the most popular Prometheus exporters on Grafana.

What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.

— prometheus.io

Why Prometheus.?

• Easy to Configure, Deploy and Maintain.
• Designed for multiple services.
• It can provide fairly simple pre-made docker images and also other pre-made configurations for similar tools like Docker.
• PromQL- a flexible query language to leverage this dimensionality, multiple modes of graphing and dashboarding support.
• Time series collection happens via a pull model over HTTP, pushing time series is supported via an intermediary gateway.
• Prometheus uses exporters that are installed and configured on the clients in order to convert and expose their metrics in a Prometheus format
• By default Prometheus comes with a UI that can be accessed on port 9090 on the Prometheus server.

Architecture of Prometheus

The Prometheus monitoring system comprises multiple components which can be used to leverage maximum benefits of the setup. The components are explained below :

Prometheus Architecture. Credits : https://prometheus.io/

• Prometheus Server : The main server that scrapes and stores the scraped metrics in a time series DB.
• Scraper : Prometheus server uses a pulling method to retrieve metrics.
• Target : The Prometheus servers clients that it retrieves info from.
• Exporter : Target libraries that convert and export existing metrics into Prometheus format.
• Alert Manager : Component responsible for handling alerts.
• Push-gateway: The Push gateway is an intermediary service which allows you to push metrics from jobs which cannot be scraped.

The Service discovery component is a feature that allows Prometheus to auto discover different targets since these targets will come and go so frequently in a distributed system or microservice orchestration style of architecture.

We will not be responsible for continuously updating a static list of target addresses each time a service and/or a piece of infrastructure is removed or added.

Setting up Prometheus

Prometheus

1. Launch an Ubuntu 20.04 server and update the package information

sudo -i
wget https://github.com/prometheus/prometheus/releases/download/v2.1.0/prometheus-2.1.0.linux-amd64.tar.gz

Note: Prometheus can be installed on earlier versions of Ubuntu as well as other operating systems. Refer to the installation guide for Windows/MacOS installations.

2. Extract the Prometheus archive

tar -xf prometheus-2.1.0.linux-amd64.tar.gz

3. Move the binaries to /usr/local/bin:

sudo mv prometheus-2.1.0.linux-amd64/prometheus prometheus-2.1.0.linux-amd64/promtool /usr/local/bin

4. Now, we need to create directories for configuration files and other Prometheus related data folders

sudo mkdir /etc/prometheus /var/lib/prometheus

5. Then, we move the configuration files to the directory we made previously

sudo mv prometheus-2.1.0.linux-amd64/consoles prometheus-2.1.0.linux-amd64/console_libraries /etc/prometheus

6. We can delete the leftover files as we do not need them any more

rm -r prometheus-2.1.0.linux-amd64*

7. We will create a systemd unit file in /etc/systemd/system/prometheus.service with the following content

[Unit]
Description=Prometheus
After=network.target
[Service]
User=root
Type=simple
ExecStart=/usr/local/bin/prometheus \
 - config.file /etc/prometheus/prometheus.yml \
 - storage.tsdb.path /var/lib/prometheus/ \
 - web.console.templates=/etc/prometheus/consoles \
 - web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target

8. We can now reload systemd

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

9. Finally, lets create a configuration file in /etc/prometheus/prometheus.yml

global:
scrape_interval: 10s
scrape_configs:
- job_name: 'prometheus_metrics'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']

Congratulations, your Prometheus server will be hosted at

http://[your_server_ip]:9090

NOTE:- If the UI is not loading please check if port 9090 is open in the machine.

Integrating Prometheus into Grafana

As we are already running the Grafana server at http://[your_server_ip]:3000 (explained in Part I of the blog), let’s direct to the Grafana UI and add Prometheus data source.

• Click on the Grafana logo to open the sidebar
• Click on “Data Sources” in the sidebar
• Choose “Add New”
• Select “Prometheus” as the data source.
• Set the Prometheus server URL (in our case: http://localhost:9090/)
• Click “Add” to test the connection and to save the new data source

Adding data source for Prometheus in Grafana

Since now we have all the components and clouds integrated at one place let’s get started with creating dashboards using the data sources.

Dashboards using Prometheus

There are a number of libraries and servers which help in exporting existing metrics from third-party systems as Prometheus metrics. This is useful for cases where it is not feasible to instrument a given system with Prometheus metrics directly (for example, HAProxy or Linux system stats).

For now we would like to show you how to install and create dashboards using two important exporters i.e. Node exporter and Black Box exporter.

Node Exporter

Node Exporter is a Prometheus exporter for hardware and OS metrics with pluggable metric collectors. It allows us to measure various machine resources such as memory, disk, and CPU utilization. Prometheus uses Node Exporter to collect the metrics of the target nodes.

Installing Node exporter

1. Login to the machine which needs to be monitored and follow the below steps

sudo -i
wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

2. Extract the node_exporter and rename the directory to ‘node_exporter’

tar -xf node_exporter-0.18.1.linux-amd64.tar.gz
mv node_exporter-0.18.1.linux-amd64 node_exporter

3. Next, we will create a new service file for the node_exporter. We go back to the root shell and navigate to the “/etc/systemd/system” directory and create new node_exporter service file ‘node_exporter.service’ using vim

cd /etc/systemd/system/
vim node_exporter.service

4. Paste the following configuration.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=root
ExecStart=/root/node_exporter/node_exporter
[Install]
WantedBy=default.target

5. Now reload the systemd service

systemctl daemon-reload

6. Then start the node_exporter service and enable it to launch every time at system startup

systemctl start node_exporter
systemctl enable node_exporter

7. Open port 9100 for the Prometheus to access

firewall-cmd — add-port=9090/tcp — permanent
firewall-cmd — reload

8. Edit /etc/prometheus/prometheus.yml and add the following

global:
scrape_interval: 10s
scrape_configs:
- job_name: 'prometheus_metrics'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'node_exporter_metrics'
scrape_interval: 5s
static_configs:
- targets: [node exporter machineIP:9100']

9. Now restart the service

sudo systemctl restart prometheus

10. Now when you click on Status => Targets on the Prometheus UI, you should see two targets — one is local host and one more is the node exporter.

We have successfully installed the node exporter.

Note: We can monitor any number of servers by following the above steps of installing the node exporter in the respective machine and adding the IP address into the Prometheus configuration file.

Creating dashboard for node-exporter in Grafana

We can create dashboards as per our requirement or we can also import in built Grafana dashboards which are provided by Grafana labs -https://grafana.com/grafana/dashboards/

We are going to import a built-in dashboard for our node exporter visualization.

• Direct to Grafana, click on + and click on import

Importing dashboard from Grafana Marketplace

• Add 1860 under import via grafana.com and click on load. This loads the Grafana dashboard from https://grafana.com/grafana/dashboards/1860
• Select the data source Prometheus and click on import.
• You should see a dashboard and get the visualization of the respective monitored VM

Node Exporter Dashboard

Black box exporter

This type of monitoring mainly refers to the monitoring state of services in the system. Using this type of monitoring we ensure things like status of the application being available or unavailable.

Prometheus black box exporter exposes the probe endpoint that takes in module and target query parameters. Default modules include configs for HTTP/S, DNS, TCP, and ICMP. For our simple uptime check, we will use the http_2xx module and configure the target to be the web endpoints we want to monitor.

Configuring Black box exporter

1. Login to the Prometheus box and edit etc/prometheus/prometheus.yml

2. Add the following code

global:
scrape_interval: 10s
scrape_configs:
- job_name: 'prometheus_metrics'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'node_exporter_metrics'
scrape_interval: 5s
static_configs:
- targets: [node exporter machineIP:9100']
- job_name: 'prometheus-blackbox-exporter'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://Simplilearn.com
- https://google.com
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance

3. Now restart the service

sudo systemctl restart prometheus

Now on Prometheus UI in the targets you should be now seeing prometheus-blackbox-exporter with the endpoints mentioned.

Creating dashboard for black box exporter in Grafana

• Direct to Grafana and click on import and enter 5345 and click on load. This loads the dashboard from https://grafana.com/grafana/dashboards/5345
• Add Prometheus data source and click on import

Blackbox Dashboard

Similarly we have multiple exporters which can be configured and monitored using Prometheus and Grafana . You can find more on https://prometheus.io/docs/instrumenting/exporters/

We have seen how to use exporters and monitor the same using Prometheus and Grafana. In the upcoming blog, we will see how to create dashboards using different cloud data sources.

Conclusion

This blog explains the set up of Prometheus and installation of popular Prometheus exporters. We were also able to visualize the metrics by creating dashboards on Grafana.

In the next blog, we will visualize metrics from the cloud providers — AWS/Azure/GCP and also create a single dashboard which displays metrics across all the cloud providers!

We would love to hear from you on what you thought about this set up in your comments. Stay tuned!