How to deploy Prometheus Helm chart to monitor Jenkins performance

— step by step guide

In this exercise, I hope to setup a prometheus/grafana dashboard to display performance metric from Jenkins.

— My Jenkins instance is deployed as a docker container;

— Separately, I installed a single node kubernetes cluster using minikube;

— I will deploy prometheus/grafana to this kubernetes cluster using Helm.

Software environment

OS:                   Ubuntu 18.04.4 LTS
Docker version:       Docker CE 19.03.8
Kubernetes version:   v.1.18
Minikube version:     v1.11.0
Helm version:         v3.2.4

First, from the Jenkins side, we need to install “prometheus metrics” plugin to stream out performance metrics as time series data.

after a while, we shall see data streams from http://prd-sal-demo01.dc2.lan:8080/prometheus/ like below,

Then we can use prometheus/grafana to visualize those metrics.

Now let’s move on to prepare environment and launch minikube,

kubectl create namespace monitoring
minikube start
helm repo add stable 
https://kubernetes-charts.storage.googleapis.com

preparing scrape_configs using values.yaml,

1. download a values.yaml template from https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml
2. locate the section "scrape_configs:" and modify as per below:
scrape_configs:
      - job_name: jenkins
        metrics_path: '/prometheus'
        static_configs:
          - targets:
            - prd-sal-demo01.dc2.lan:8080

install prometheus chart using helm with above values.yaml,

$helm install  prometheus-demo01  --namespace monitoring -f values.yaml stable/prometheus
NAME: prometheus-demo01
LAST DEPLOYED: Fri Jun 26 08:52:34 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-demo01-server.monitoring.svc.cluster.local
Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9090
The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-demo01-alertmanager.monitoring.svc.cluster.local
Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Pod Security Policy has been moved to a global property.  #####
######            use .Values.podSecurityPolicy.enabled with pod-based      #####
######            annotations                                               #####
######            (e.g. .Values.nodeExporter.podSecurityPolicy.annotations) #####
#################################################################################
The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-demo01-pushgateway.monitoring.svc.cluster.local
Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9091
For more information on running Prometheus, visit:
https://prometheus.io/

get the pod name of the prometheus server pod,

export pPOD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")

map host port 9912 to prometheus server pod port 9090,

kubectl --namespace monitoring --address 0.0.0.0 port-forward $pPOD_NAME 9912:9090

test from CLI first, ok.

visit prometheus at host port 9912 from browser; yes, prometheus is up.

Moving on to Grafana,

helm install grafana-demo01 — namespace monitoring stable/grafana
export gPOD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana-demo01" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $gPOD_NAME 9911:3000

Now grafana instance is up; let’s add prometheus as a data source to grafana,

Test connection to data source, ok.

download a pre-configured Grafana dashboard template as json from https://grafana.com/grafana/dashboards/9964, and import it to grafana.

yes, we now see my data in Grafana dashboard.

putting everything together,

# reset environment to start refresh
minikube delete
minikube start
kubectl create namespace mon
helm ls --namespace mon
# delete all helm releases
helm ls --all --short --namespace mon | xargs -L1 helm delete --namespace mon
# install prometheus and grafana using Helm
helm install  prometheus-demo01  --namespace mon -f values.yaml stable/prometheus
helm install grafana-demo01 --namespace mon stable/grafana
# wait for a while
# port forward for prometheus pod
export pPOD_NAME=$(kubectl get pods --namespace mon -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
echo $pPOD_NAME
kubectl --namespace mon --address 0.0.0.0 port-forward $pPOD_NAME 9912:9090 &
# port forward for grafana pod
export gPOD_NAME=$(kubectl get pods --namespace mon -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana-demo01" -o jsonpath="{.items[0].metadata.name}")
echo $gPOD_NAME
kubectl get secret --namespace mon grafana-demo01 -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
kubectl --namespace mon --address 0.0.0.0 port-forward $gPOD_NAME 9911:3000 &

Notes:

if you can’t get the values.yaml work or you wish to change the config on the fly, you can modify configmap with:

kubectl edit configmap prometheus-demo01-server — namespace monitoring -o jenkins.yml

insert to section scape_configs:

wait for a while, a lot of jenkins data pops up,

jobs status

you can also plot a simple graph from prometheus.

no. of plugins in jenkins instance

remember to add in ‘ — address’ when you do port forwarding; by default, it is only allowed for localhost access, which means you can not access it from browser.

it takes a while for the pod to be properly initialized.

if you used a non-default namespace, remember to add the ‘ — namespace’ option to all your cli.

if you are troubleshooting the ports, use the following to make sure the port is not already bind to something else:

alias listen='lsof -PiTCP -sTCP:LISTEN'
>listen |grep $port

helm install syntax has changed from helm 2 to helm 3; with helm 2, you can use ‘ — name’ to define helm release name.