Monitoring Linux Processes using Prometheus and Grafana

Antoine Solnichkin
May 18, 2019 · 13 min read
Image for post
Image for post
Image for post
Image for post

What You Will Learn


Unix Process Monitoring Basics

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

Detailing Our Monitoring Architecture

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

Installing The Different Tools

a — Installing Pushgateway

wget https://github.com/prometheus/pushgateway/releases/download/v0.8.0/pushgateway-0.8.0.linux-amd64.tar.gz
> tar xvzf pushgateway-0.8.0.linux-amd64.tar.gz 
> cd pushgateway-0.8.0.linux-amd64/
> ./pushgateway &
me@schkn-ubuntu:~/softs/pushgateway/pushgateway-0.8.0.linux-amd64$ ./pushgateway & 
[1] 22806
me@schkn-ubuntu:~/softs/pushgateway/pushgateway-0.8.0.linux-amd64$ INFO[0000] Starting pushgateway (version=0.8.0, branch=HEAD, revision=d90bf3239c5ca08d72ccc9e2e2ff3a62b99a122e) source="main.go:65"INFO[0000] Build context (go=go1.11.8, user=root@00855c3ed64f, date=20190413-11:29:19) source="main.go:66"INFO[0000] Listening on :9091. source="main.go:108"

b — Installing Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.9.2/prometheus-2.9.2.linux -amd64.tar.gz
> tar xvzf prometheus-2.9.2.linux-amd64.tar.gz 
> cd prometheus-2.9.2.linux-amd64/
> vi prometheus.yml
global:
scrape_interval: 1s # Set the scrape interval to every 1 second.
static_configs:
- targets: ['localhost:9090', 'localhost:9091']
Image for post
Image for post

c — Installing Grafana

> wget https://dl.grafana.com/oss/release/grafana_6.2.0-beta1_amd64.deb> sudo dpkg -i grafana_6.2.0-beta1_amd64.deb
> sudo systemctl status grafana-server
● grafana-server.service - Grafana instance
Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2019-05-09 10:44:49 UTC; 5 days ago
Docs: http://docs.grafana.org
Image for post
Image for post
Image for post
Image for post

Building a bash script to retrieve metrics

Image for post
Image for post
Image for post
Image for post
> touch better-top 
> chmod u+x better-top
> vi better-top
#!/bin/bash
z=$(ps aux)
while read -r z
do
var=$var$(awk '{print "cpu_usage{process=\""$11"\", pid=\""$2"\"}", $3z}');
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/machine
> while sleep 1; do ./better-top; done;
Image for post
Image for post
Image for post
Image for post

Building An Awesome Dashboard With Grafana

Image for post
Image for post

1 — Building Rounded Gauges

Image for post
Image for post

a — Retrieving the current overall CPU usage

Image for post
Image for post

b — Retrieving the average CPU usage

Image for post
Image for post

2 — Building Horizontal Gauges

Image for post
Image for post

3 — Building Vertical Gauges

Image for post
Image for post

4 — Building Line Graphs

Image for post
Image for post
Image for post
Image for post

Bonus : explore data using ad hoc filters

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

A quick word to conclude


devconnected — DevOps, Sysadmins & Engineering

Tutorials & Guides for DevOps, sysadmins and software…

Antoine Solnichkin

Written by

Software Engineer | Technical Writer @ devconnected | Into DevOps — System Administration — Open Source http://devconnected.com

devconnected — DevOps, Sysadmins & Engineering

Tutorials & Guides for DevOps, sysadmins and software engineers.

Antoine Solnichkin

Written by

Software Engineer | Technical Writer @ devconnected | Into DevOps — System Administration — Open Source http://devconnected.com

devconnected — DevOps, Sysadmins & Engineering

Tutorials & Guides for DevOps, sysadmins and software engineers.