Deploying Prometheus in Kubernetes with Persistent disk or ConfigMap

Sujith R Pillai

4 min readMay 11, 2020

This blog walk you through the steps to create a Prometheus instance in Kubernetes with Persistent disk.

Why should I do this ?

Do you have any of the below listed requirements, then you can use this blog for help,

Implement a monitoring platform for your kubernetes application
Monitor a “Blockchain Application” (Refer my blog here)
Want to persist data for your prometheus implementation

Pre-requisite

You need to have a Kubernetes cluster . If you don’t have one, you can easily create one with the following steps listed here.

Step 1: Create a namespace

If you already have one namespace dedicated for monitoring, create one. This is not mandatory, however it helps you to manage your application effectively.

Create a file with the namespace configuration,

cat <<EOF > monitoring-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
EOF

Apply the configuration to create the namespace,

kubectl apply -f monitoring-ns.yaml

Step 2 : Create necessary roles, accounts and permissions

Create a file clusterRole.yaml with the following content,
(A sample is available here)

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
  namespace: monitoring
rules:
- apiGroups: [""]
  resources:
  - nodes
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

Create a file serviceAccount.yaml with the following content,
(A sample is available here)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring

Create a file clusterRoleBinding.yaml with the following content,
(A sample is available here)

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
  namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

Apply the configurations,

kubectl apply -f clusterRole.yaml
kubectl apply -f serviceAccount.yaml
kubectl apply -f clusterRoleBinding.yaml

Step 3: Create Prometheus scrape configuration

Create a file with name prometheus.yml

global:
  scrape_interval: 15s
# Scraping Prometheus itself
scrape_configs:
- job_name: 'Prometheus'
  static_configs:
  - targets: ['prometheus-service:9090']

NOTE: You will need to edit this file to add more scrape configurations.

Important : Use only one of the options in Step 4 (Either Option 1 or Option 2)

Step 4 : Option 1: Create Persistent Volume

If you want to preserve the Prometheus configuration, use a persistent volume. In this step, I am explaining how to use a “Local Filesystem” directory as persistent volume. You may use any other types such as NFS, GlusterFS, Ceph, Public cloud storage…etc.

Create a directory with necessary permissions,

mkdir -p /persistentvolume/prometheus
chmod 755 -R /persistentvolume

Create a persistent volume configuration file prometheus-pv-volume.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-pv-volume
  namespace: monitoring
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 256Mi 
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/persistentvolume/prometheus"

Create a persistent volume claim configuration file prometheus-pv-claim.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-pv-claim
  namespace: monitoring
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi

Apply the configuration,

kubectl apply -f prometheus-pv-volume.yaml
kubectl apply -f prometheus-pv-claim.yaml

Copy the prometheus configuration file created in Step 3 to the persistent volume directory

cp prometheus.yml /persistentvolume/prometheus/

Step 4 : Option 2: Create ConfigMap

Create a configmap with the configuration file created in Step 3,

kubectl create configmap prometheus-config --from-file=./prometheus.yml -n monitoring

Step 5: Deploy Prometheus

Create a file prometheus-deployment.yaml with the below configuration (Use either Option 1 or Option 2, depending on the Option that you selected in Step 4)

Option 1 : If using Persistent Volume,

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus-cont
        image: prom/prometheus
        volumeMounts:
        - name: prometheus-volume
          mountPath: /etc/prometheus/
        ports:
        - containerPort: 9090
      volumes:
      - name: prometheus-volume
        persistentVolumeClaim:
          claimName: prometheus-pv-claim
      serviceAccountName: prometheus

Option 2: If using ConfigMap,
(A sample is available here)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus-cont
        image: prom/prometheus
        volumeMounts:
        - name: config-volume
          mountPath: /etc/prometheus/prometheus.yml
          subPath: prometheus.yml
        ports:
        - containerPort: 9090
      volumes:
      - name: config-volume
        configMap:
          name: prometheus-config
      serviceAccountName: prometheus

Step 6: Expose the application

Create a file prometheus-service.yaml with the following content,
(A sample is available here)

kind: Service
apiVersion: v1
metadata:
  name: prometheus-service
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
  - name: promui
    nodePort: 30900
    protocol: TCP
    port: 9090
    targetPort: 9090
  type: NodePort