Monitoring Your Server Logs with Grafana & Loki & Promtail.
Hello learners !!
Today, I want to talk about the concept of observability and how it can be achieved using Grafana & Promtail & Loki. If you are new to Grafana or Loi & promtail, don’t worry! This blog will provide you with a hands-on tutorial to help you understand and implement observability in your applications.
Working of this project is simple:
Grafana will ask logs data from Loki which is an Loki is an open-source, multi-tenant log aggregation system & Loki will ask data from Promtail which work is to collect log data from different source.
So let’s start making a hands-on project…
Now search ec2 in search bar, go to instance, click on launch instance. Give you instance a name. Here i have given “Grafana demo”.
Select Ubuntu image & free tier. Select no key pair and allow all traffic & then click on Launch instance.
Now select your instance & click on connect. Again click on connect to access CLI of our server.
Now run the following commands to install Grafana on our Ubuntu server:
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key
https://apt.grafana.com/gpg.key
Now we will get key of a stable version of Grafana.
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
No update the system.
sudo apt-get update
After updating, we will install our devops tools Grafana.
sudo apt-get install grafana
After installing grafana, let’s start grafana server & enable it so after every start it will start automatically.
sudo /bin/systemctl start grafana-server
sudo /bin/systemctl enable grafana-server
Now select your instance, go to security tab, click on your security group. After that click on “Edit Inbound rule”.
Now click on add rule and give port number ‘3000' open to your grafana server & select my ip as of security reasons. Click on add rule.
Copy your instance public IP, paste in browser:
publicip:3000
By default its username & password is Admin:Admin. After that you will ask to set up your own password.
Here’s is your Grafana dashboard running on your ec2 server.
Now install Docker as we will install Loki & promtail from Docker image.
sudo apt-get install docker.io
Add user as docker need a user to access images.
sudo usermod -aG docker $USER
sudo reboot
docker ps
Now make a directory in which we will download our Loki & promtail yaml files to create docker images & container.
mkdir grafanafolder
cd grafanafolder
Run below command to donload yaml file for Loki.
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
ls
Run below command to donload yaml file for Promtail.
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
ls
Dokcer command to build image for Loki.
docker run -d — name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 — config.file=/mnt/config/loki-config.yaml
docker ps
Now we need to allow ‘3001' port to access Loki on our server. Go to your instance again, select your instance, go to security tab, select security group, Click on edit inbound rule.
Give port 3100 and select myip as for security reasons. Click on add rule.
You can access Loki by pasting below url:
publicipaddress:3100/ready
Now run docker command to build image for promtail image & link with that with Loki docker image so it can access logs.
docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml
After build both images, now go to grafana dashboard, click on add DataSource & there click On Loki. In Loki paste below url in URL block:
http://localhoat:3001
As our Loki is running in local server in our Main server, thats why we use localhost in place of ip. Now click on save& Test. This will add data source.
After that click on explore view.
Now select in label filter as “Job” & “verlogs” and click on “run query”. This will show all logs of system.
Now click on Add dashboard. This will add our logs into our dashboard.
Now go to Add >> Visualization.
First come to your instance CLI, here we will install nginx server and show how many times nginx has been there in logs as visualization.
To install Nginx server, run below command:
sudo apt-get install nginx
After installing, coming back to grafana dashboard. Now select in label filter “jobs” & “verlogs”. In operation tab select rate.
That’s it. Run query button. graph will show. you can even customize graph view as per your choice.
Now click on Apply. Your grafana dashboard is ready.
Don’t forgot to click on Save dashboard. And even you can start auto refresh too.
Here is your Final Grafana monitoring our server dashboard is here.
Thanks for reading !!!
For any issue, contact me :
