Step-by-Step to a Seamless Monitoring Setup: Prometheus and Grafana (Part 1)

In today’s world, Monitoring is very important. We need to monitor our servers to understand their performance. It’s essential to track CPU, memory, disk, network, and I/O metrics in some specific situations to improve system performance. Additionally, we may also need to know how many requests are coming to our servers, the average response time, and the number of errors occurring.

In this article, we will see how to set up a seamless monitoring environment for our servers using Prometheus and Grafana.

I am going to divide this topic (Step-by-Step to a Seamless Monitoring Setup: Prometheus and Grafana) into three parts. In the first part, we will see how to set up Prometheus and Grafana using Docker and how to create dashboards in Grafana. In the next part of this blog, we will explore how to set up Alertmanager, create alerts for our servers, and configure a notification channel to send alerts to our Slack Channel. In the final part, I will show you How to secure your local Prometheus Server and Node Exporter using Basic Authentication and SSL. Although this is not related to this workflow, it will help you gain knowledge of securing your Prometheus Server.

Before we start, let’s understand why we need monitoring.

Why we need monitoring?

Picture Credit: Meta AI

Let’s say we have a application. That is either going to run on physical server or cloud server. Any way it’s going to run on a server. Even if our application runs on Serverless, it’s going to run on a server. Every application will produce some logs. Logs are the information about the application. It can be error, warning, info, debug etc. Logs are going to be logged in log storage database. Some of the log storage databases are Elasticsearch, Splunk, Kibana etc. Logs are useful when some event was already happened and we know how to diagnose the issue. we can read the log files and can fix the problem.

But What if we want to notified before the event happens. Also we want more data just than logs. We want to know the performance of the application. We want to know the CPU, Memory, Disk, Network, I/O etc. Or Let’s say we have to identify why our application is slow?

Some of the Example Scenarios:

• How many users are using our application from particular region?
• How many errors and exceptions are happening in our application per day?
• How many requests are coming to our application per day?
• What is the average response time of our application?
• How many servers our application is using, if we are using cloud?

Let’s start,

We are going to have two servers for this demo. One is going to run Prometheus and Grafana and the other one will be like the server which we are going to monitor. In this article, I am going to use Ubuntu 20.04 servers on AWS Cloud

Steps:

Step 1: Launching an Ubuntu Instance:

In my case, I don’t have Linux machines. So I am going to launch two Ubuntu instances on AWS.

If you also don’t have one, try to launch a Linux machines using cloud providers, VirtualBox, or any other method of your choice.

If you look at the above picture I launched two servers. One is Prometheus and Grafana server and the other one is the server which we are going to monitor. I launched t2.large for Prometheus and Grafana server and t2.micro for the server which we are going to monitor. You can even use t2.micro for Prometheus and Grafana server as well but it will be little slow. I’m using t2.large because I’m going to run both Prometheus and Grafana on the same server. If you want to run Prometheus and Grafana on different servers, you can use t2.micro for Prometheus and Grafana servers as well.

Don’t forget to open necessary Ports on your EC2 Instance Security Group.

Step 2: Setting up Prometheus and Grafana using Docker

There are numerous ways to set up Prometheus and Grafana. We can do it manually by downloading the binaries and running them on our server.

But the easiest way is to use Docker. In this article I’ll show you the easiest way to set up Prometheus and Grafana using Docker.

In case if you want to set up manually, you can follow the below repository link to set up Prometheus and Grafana.

For setting up manually: Click here

Let’s start by installing Docker first on Ubuntu server where we want to run our Prometheus and Grafana .

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo systemctl enable docker
sudo usermod -aG docker ubuntu

Once Docker is installed, we can proceed with setting up our Prometheus and Grafana Servers.

• First, let’s set up Prometheus. Since we are going to use `Docker` to set up Prometheus, we need to create a mount point for the Prometheus configuration file because we cannot go each and every time to the container to change the configuration file. So, let’s create a directory and create a new file called prometheus.yml .
• Create a directory /etc/prometheusand create a new file called prometheus.yml

mkdir /etc/prometheus
vim prometheus.yml

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

Once done, let’s start our Prometheus server using Docker. Run the below docker command to start the Prometheus server

docker run -d - name=prometheus -e TZ=IST -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -p 9090:9090 ubuntu/prometheus:2.48.0–22.04_stable

The above command will start a Prometheus container with name prometheus and it will expose the Prometheus web interface on port 9090. You can access the Prometheus web interface by visiting http://<node-ip>:9090 in your web browser.

• Now, let’s set up Grafana. For Grafana we are not going to work with any configuration file. We can directly start the Grafana container using the below command.

docker run -d - name=grafana -p 3000:3000 grafana/grafana

Now, open your browser and navigate to http://<node-ip>:3000. You will be prompted to log in. The default username and password are both admin. Once you log in, you will be asked to change the password. You can change the password or skip it.

Step 3: Connecting Prometheus and Grafana

Now we are going to add Prometheus as a data source for Grafana. You can follow the steps below to connect Prometheus and Grafana.

Step 4: Setting up the Server to be monitored

Consider we have a application running on a server and we want prometheus to monitor the server. If we want our Application to send metrics to prometheus, we need to have to access the application code and modify it to send metrics to prometheus. Because the code is running on the server and we have to modify the code to send metrics to prometheus. But what if we don’t have access to the application code? or we can’t change the application code?

Example: If we have some external mediums and we want to monitor them, like MySQL Database we can’t change the code of MySQL to send metrics to prometheus. In such cases, we can use Exporters.

Exporters

Exporters are something we can use with prometheus to collect the metrics from the server or application and expose them to prometheus. Exporters are component of Prometheus.

How Exporters work?

We need to install the Exporter on the medium we want to monitor or next to that medium.

For example, If we want to monitor Linux server, we have to install Node Exporter on the server. If we want to monitor MySQL, we have to install MySQL Exporter next to the MySQL.

Once we install the Exporter, it will collect the metrics from the medium and expose them to prometheus. Prometheus will connect to Exporter and scrape the metrics from the Exporter. This process is called Scraping. By default, Prometheus scrapes the metrics from the Exporter every 15 seconds. We can define our own scraping configurations in the prometheus.yml file.

Node Exporter

Node Exporter is a Prometheus exporter for collecting metrics from Unix systems. Node exporter is one of the official exporters of Prometheus. It provides a detailed report of the system’s resources like CPU, memory, disk, network and more. we can also extend the Node Exporter functionality by using pluggable metric collectors.

Installation of Node Exporter:

Now we need to SSH into the server we want to monitor. After that, follow the steps below to install Node Exporter on the server to expose metrics:

1. Go to the Node Exporter download page.
2. Copy the link address of the node exporter tar file. Select the appropriate link based on your OS like Darwin for MacOS, Linux for Linux OS.
3. Open the terminal and run the below command to download the Node Exporter tar file.

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.1/node_exporter-1.8.1.linux-amd64.tar.gz

4. Unzip the downloaded tar file.

tar xvf node_exporter-1.8.1.linux-amd64.tar.gz

5. Change the directory to extracted directory.

cd node_exporter-1.8.1.linux-amd64

6. Execute node exporter binary file.

./node_exporter

Now the Node Exporter is running it will expose the metrics on the port 9100. We can access the metrics by visiting the URL http://<node-ip>:9100/metrics.

Step 5: Updating the Scraping Configuration in Prometheus Server

Now we have the Node Exporter running on our Linux server. If we want our Prometheus server to scrape the metrics from the Linux server, we need to update the scraping configuration in the prometheus.yml file.

1. Open the prometheus.yml file.

cd /etc/prometheus
vi prometheus.yml

2. Under the scrape_configs section, add the below configuration. By default, You'll have some configuration in the scrape_configs section that is the configuration for scraping the metrics from Prometheus itself. Add the below configuration to scrape the metrics from the Node Exporter.

- job_name: 'node-exporter'
    static_configs:
    - targets: ['<node-ip>:9100']

3. Restart your Prometheus docker container to apply the Changes.

Now we need to ensure that our Prometheus server can scrape metrics from the Node Exporter, To Verify this go to Prometheus Server and the Click on “Status” → Click on “Targets”

Step 8: Creating Dashboards to Monitor CPU Usage of Our Servers

One Option:

We can visualize our metrics using the Graph tab in the Prometheus web UI. The graph visualization helps us understand the patterns of our metrics over time, such as whether they are increasing, decreasing, or remaining constant.

To monitor the CPU usage of our server, we can use a simple query that shows the CPU usage in seconds. You can use this query or modify it to suit your needs:

Note: The graph visualization is only available for the Instant Vector data type not for the Range Vector data type.

Second Option(Best Option):

Although Prometheus comes with some basic visualization capabilities. But the visualization capabilities of Prometheus are not so good. If we need to visualize our metrics in a more shopisticated way, prometheus is not a good option. In this case we can use Grafana to visualize our metrics in a more sophisticated way.

Now, navigate to your Grafana Server. And then follow the below steps.

That’s all about the first part of setting up Prometheus and Grafana using Docker and how to create dashboards in Grafana. I will try to post the next part tomorrow or the next day.

I have created a repository for this full setup from Part 1 to Part 3. You can find all the configuration files related to all the parts there.

Repository Link: Click here

👏 If you find this helpful, don’t forget to give claps and follow my profile. I’ll be sharing more projects and ideas about Cloud and DevOps. If you have any doubts, feel free to comment or message me on LinkedIn.

Let’s Connect on LinkedIn: LinkedIn Profile

Explore Hands-On Projects: My GitHub Account