Prometheus & Grafana with Persistent Storage

Integrate Prometheus and Grafana and perform in following way:

1. Deploy them as pods on top of Kubernetes by creating resources Deployment, ReplicaSet, Pods or Services

2. And make their data to be remain persistent

3. And both of them should be exposed to outside world

Integrating Prometheus and Grafana with the container management tool Kubernetes. This makes the Prometheus and Grafana containers easily manageable. Prometheus needs to be provided with some node exporter software so that Prometheus can collect the metrics whenever required from this exporter software installed inside various operating systems. Run the below mentioned commands to download and start the node exporter software inside the operating system.

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.0/node_exporter-1.0.0.rc.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.0.0.rc.1.linux-amd64.tar.gz
cd node_exporter-1.0.0.rc.1.linux-amd64
./node_exporter

The downloaded files of the Prometheus are copied into a separate folder so that they can be edited or managed accordingly without any error. For creating the containers of Prometheus , create a docker image using Dockerfile so that whenever the container has been launched , Prometheus services launch inside the container Use the following commands to create the Prometheus image.

ROM centos
RUN yum install wget tar -y
RUN wget https://github.com/prometheus/prometheus/releases/download/v2.19.2/prometheus-2.19.2.linux-amd64.tar.gz
RUN mkdir /yashprom
RUN tar -xzf prometheus-2.19.2.linux-amd64.tar.gz
RUN cp -rf prometheus-2.19.2.linux-amd64/* /arnav
RUN cp /arnav/prometheus /usr/bin
CMD prometheus — config.file=/arnav/prometheus.yml

Use the commands to build the image and push to the docker hub if required.

docker build -t (name of image) (path of Dockerfile)
docker push (image name)

Similarly create the Grafana image following the below mentioned commands:

FROM centos
RUN yum install wget -y
RUN wget https://dl.grafana.com/oss/release/grafana-7.0.6-1.x86_64.rpm
RUN yum install grafana-7.0.6–1.x86_64.rpm -y
WORKDIR /usr/share/grafana
ENTRYPOINT [“/usr/sbin/grafana-server”, “ — config=/etc/grafana/grafana.ini”, “cfg:default.paths.logs=/var/log/grafana”, “cfg:default.paths.data=/var/lib/grafana”, “LimitNOFILE=10000”]

It is necessary to provide the details of nodes that are to be managed inside the configuration file of the Prometheus. Here the role of ConfigMap resource of the Kubernetes becomes useful. The details of the nodes that are to be monitored are mentioned inside this ConfigMap file as given below: ( Here I took 2 nodes with so mentioned IP addresses with the node exporter services running inside the nodes)

kind: ConfigMap
apiVersion: v1
metadata:
name: prometheus-nodes
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets:
rule_files:
scrape_configs:
- job_name: ‘rhelnode’
static_configs:
- targets: [‘192.168.225.80:9100’]
- job_name: ‘clinode’
static_configs:
- targets: [‘192.168.225.124:9100’]

Now it is up to creating the Kubernetes resources for the Prometheus and Grafana ;

It is required to provide persistent storage to the Prometheus pod so as to keep the data permanent. Also it is mandatory to attach the ConfigMap resource to this pod. Finally the deployment service is provided to the pod so that there is no tension in the downtime or unexpected deletion of the pods. This deployment is exposed so that it can be accessed by the admins. This whole setup can be created by using a single YAML file.

apiVersion: v1
kind: Service
metadata:
name: prometheus
spec:
selector:
env: deploy
type: NodePort
ports:
- port: 9090
targetPort: 9090
 — -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
 — -
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
labels:
env: deploy
spec:
replicas: 1
selector:
matchLabels:
env: deploy
template:
metadata:
name: prometheus-pod
labels:
env: deploy
spec:
containers:
- name: promcont1
image: arnav/prometheus:x2
volumeMounts:
- name: logs
mountPath: /yashprom
volumeMounts:
- name: ymlnodes
mountPath: /arnav/prometheus.yml
subPath: prometheus.yml
volumes:
- name: logs
persistentVolumeClaim:
claimName: prometheus-pvc
- name: ymlnodes
configMap:
name: prometheus-nodes
defaultMode: 0744

YAML file for Grafana. Provide the PVC to make the data persistent and expose the deployment to make the pod accessible and there is no room for downtime or unexpected failures.

apiVersion: v1
kind: Service
metadata:
name: grafana
spec:
selector:
env: deploy
type: NodePort
ports:
- port: 3000
targetPort: 3000
 — -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
 — -
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
spec:
replicas: 1
selector:
matchLabels:
env: deploy
template:
metadata:
name: grafana-pod
labels:
env: deploy
spec:
containers:
- name: g1
image: arnav/grafana:x1
volumeMounts:
- name: data
mountPath: /var/lib/grafana
volumes:
- name: data
persistentVolumeClaim:
claimName: grafana-pvc

Creating using these YAML files-

kubectl create -f [YAML file name]

With the whole setup deployed on top of the private Kubernetes cluster , now its the time for testing

The configured two nodes are up and data is successfully retrieved by the Prometheus. Even after the pod is deleted , since it is a deployment service of the Prometheus the deleted pods are automatically launched by the K8s with the data remaining persistent , older data can be accessed.

Moving to Grafana , initially creating a dashboard inside with a simple prom query

Now onto prometheus, enter url given by k8s.

Moving on to graphana and adding data source.

Images are explaining the steps.

Creating a basic graph in Grafana

Testing whether the data is persistent or not

Deleting the grafana pod using following command-

 kubectl delete pods grafana-7b7dc95846

As the service used is deployment , the deleted pod is again up.

Even after deleting the pod, the data is persistent and hence the practical is successful.