CODEX

Setup Kuberhealthy with Prometheus and Grafana on Minikube

Rocky Chen

Follow

Jan 10 · 9 min read

I browsed lots of online resources and instructions to setup Kuberhealthy with Prometheus and Grafana together on a local Kubernetes environment like Minikube, but unfortunately, none of them works for me.

Most of the instructions and articles listed installation steps for each of them but seldom showed the steps for:

• How to connect Kuberhealthy and Prometheus? That means how to configure Kuberhealthy service to send metrics to Prometheus service successfully.
• How to configure Grafana dashboard to display the Kuberhealthy metrics collected by Prometheus.

There are some extra steps needed for most cases after you installed them and this post would show you how to make them work together.

All of these tools are running on macOS (10.15.7).

1. Minikube Setup

Install Minikube

To install Minikue on Mac is easy by using brew:

brew install minikube

Start a Cluster

Once Minikube gets installed successfully, you could start a local Kubernetes cluster quickly with it.

I created a local cluster with two nodes with the following command:

$ minikube start -n 2
😄  minikube v1.16.0 on Darwin 10.15.7
✨  Automatically selected the docker driver. Other choices: hyperkit, virtualbox
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=4050MB) ...
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass👍  Starting node minikube-m02 in cluster minikube
🤷  docker "minikube-m02" container is missing, will recreate.
🔥  Creating docker container (CPUs=2, Memory=4050MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.49.2
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
    ▪ env NO_PROXY=192.168.49.2
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

From the output, we could learn the version for Minikube v1.16.0 and Kubernetes v1.20.0.

Then, run a kubectl command to verify it looks good:

$ kubectl get nodes
NAME           STATUS   ROLES                  AGE     VERSION
minikube       Ready    control-plane,master   3m11s   v1.20.0
minikube-m02   Ready    <none>                 2m9s    v1.20.0

2. Prometheus + Grafana Setup

Install kube-prometheus-stack

I choose the kube-prometheus-stack because it includes both Prometheus Operator and Grafana dashboards so I don’t need to set up them separately.

kube-prometheus stack, a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

Let’s install it in a separate namespace monitoring for kube-prometheus-stak with helm.

$ kubectl create namespace monitoring
$ kubectl config set-context --current --namespace=monitoring

Before installation, make sure you’ve added the repo to helm:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install prometheus prometheus-community/kube-prometheus-stack

Check if the Prometheus and Grafana installed correctly:

$ kubectl get svc -n monitoring
NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                     ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   2m49s
prometheus-grafana                        ClusterIP   10.99.119.38     <none>        80/TCP                       2m50s
prometheus-kube-prometheus-alertmanager   ClusterIP   10.102.243.73    <none>        9093/TCP                     2m50s
prometheus-kube-prometheus-operator       ClusterIP   10.106.211.104   <none>        443/TCP                      2m50s
prometheus-kube-prometheus-prometheus     ClusterIP   10.101.155.118   <none>        9090/TCP                     2m51s
prometheus-kube-state-metrics             ClusterIP   10.110.98.157    <none>        8080/TCP                     2m50s
prometheus-operated                       ClusterIP   None             <none>        9090/TCP                     2m48s
prometheus-prometheus-node-exporter       ClusterIP   10.98.183.105    <none>        9100/TCP                     2m50s

Verify Prometheus

To verify Prometheus working as expected, we could expose its webpage by minikube service command below:

$ minikube service prometheus-kube-prometheus-prometheus -n monitoring

The webpage is opened in your default browser like:

Verify Grafana Dashboards

Let’s access Grafana by exposing its service with the Minikube command:

$ minikube service prometheus-grafana -n monitoring

You would see the login page of Grafana opened in your default browser and fill in the credential of username: admin and password: prom-operator.

When going to the Dashboard->Manage page, there is a list of the pre-set dashboard for you.

This is another reason for me to choose kube-prometheus-stack to install.

3. Kuberhealthy Setup

Install Kuberhealthy

Once Prometheus and Grafana are ready in the Minikube cluster, Kuberhealthy can be installed via helm as well.

Typically, Kuberhealthy is installed in the kuberhealthy namespace.

$ kubectl create ns kuberhealthy
$ kubectl config set-context --current --namespace=kuberhealthy

Since kube-prometheus-stack uses Prometheus Operator, Kuberhealthy should also be installed for Prometheus Operator. It would install a Kuberhealthy ServiceMonitor which can be discovered by Prometheus Operator, and then Kuberhealthy is possible to send the metrics to Prometheus.

$ helm repo add kuberhealthy https://comcast.github.io/kuberhealthy/helm-repos$ helm repo update$ helm install kuberhealthy kuberhealthy/kuberhealthy --set prometheus.enabled=true,prometheus.enableAlerting=true,prometheus.serviceMonitor=true

Verify Kuberhealthy Ready

• First, there should be a kuberhealthy service in the kuberhealthy namespace.

$ kubectl get svc
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kuberhealthy     ClusterIP   10.102.231.49   <none>        80/TCP    91s

• There should 3 khchecks installed by default in the kuberhealthy namespace. khcheck is a custom resource created by Kuberhealthy in the cluster to run tests. For details, please read here.

$ kubectl get khchecks
NAME                  AGE
daemonset             3m5s
deployment            3m5s
dns-status-internal   3m5s

• The most important, ensure kuberhealthy checks able to send results to kuberhealthy service. It could be done by checking the Pod status. The pods related to khchecks should be Completed status.

+ kubectl get pods
NAME                             READY   STATUS      RESTARTS   AGE
check-reaper-1610257860-778w5    0/1     Completed   0          6m22s
check-reaper-1610258040-5vjjs    0/1     Completed   0          3m21s
check-reaper-1610258220-mg4lb    0/1     Completed   0          21s
daemonset-1610257533             0/1     Completed   0          11m
deployment-1610257533            0/1     Completed   0          11m
deployment-1610258132            0/1     Completed   0          109s
dns-status-internal-1610257652   0/1     Completed   0          9m49s
dns-status-internal-1610257772   0/1     Completed   0          7m49s
dns-status-internal-1610257892   0/1     Completed   0          5m49s
dns-status-internal-1610258012   0/1     Completed   0          3m49s
dns-status-internal-1610258132   0/1     Completed   0          109s
kuberhealthy-7b5b54ddb7-p7c7s    1/1     Running     0          12m
kuberhealthy-7b5b54ddb7-xwj9j    1/1     Running     0          12m

If the status of Pods is not Completed, please refer to one of my previous posts: Minikube NATs Source IP Address of Pods for services with ClusterIP

4. Connect Kuberhealthy to Prometheus

Most instructions end by step 3 after installing all the components, but actually, they are not working together yet. Because all the settings and configurations are in default values which might not match each other.

Check If Prometheus Receives Kuberhealthy Metrics

Let’s check if Prometheus can receive Kuberhealthy metrics first.

• Goto Service Discovery page in Prometheus to check if Kuberhealthy is on the list. By default, Kuberhealthy service is not listed.

• On the Graph page, check if Kuberhealthy metrics included. In my example, they are not included.

Therefore, Prometheus is unable to receive Kuberhealthy metrics yet, and of course, Grafana doesn’t have Kuberhealthy data either.

Why does it happen?

First, let’s have a look at ServiceMonitors in all namespaces because Prometheus Operation collecting metrics via them.

$ kubectl get servicemonitors -A
NAMESPACE      NAME                                                 AGE
kuberhealthy   kuberhealthy                                         29m
monitoring     prometheus-kube-prometheus-alertmanager              3h35m
monitoring     prometheus-kube-prometheus-apiserver                 3h35m
monitoring     prometheus-kube-prometheus-coredns                   3h35m
monitoring     prometheus-kube-prometheus-grafana                   3h35m
monitoring     prometheus-kube-prometheus-kube-controller-manager   3h35m
monitoring     prometheus-kube-prometheus-kube-etcd                 3h35m
monitoring     prometheus-kube-prometheus-kube-proxy                3h35m
monitoring     prometheus-kube-prometheus-kube-scheduler            3h35m
monitoring     prometheus-kube-prometheus-kube-state-metrics        3h35m
monitoring     prometheus-kube-prometheus-kubelet                   3h35m
monitoring     prometheus-kube-prometheus-node-exporter             3h35m
monitoring     prometheus-kube-prometheus-operator                  3h35m
monitoring     prometheus-kube-prometheus-prometheus                3h35m

The ServiceMonitor of Kuberhealthy is the top of the list which means Kuberhealthy follows Prometheus’ pattern to provide a ServiceMonitor to send metrics.

But please notice that it is in the kuberhealthy namespace instead of monitoring namespace.

Another check is for Prometheus. Prometheus here is another custom resource definition (CRD) created by Prometheus Operator.

$ kubectl get Prometheus -A
NAMESPACE    NAME                                    VERSION   REPLICAS   AGE
monitoring   prometheus-kube-prometheus-prometheus   v2.22.1   1          3h41m

We could display its configuration via kubectl command. Here is a part of its configuration in YAML:

$ kubectl get Prometheus prometheus-kube-prometheus-prometheus -n monitoring -o yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
  name: prometheus-kube-prometheus-prometheus
  namespace: monitoring
...
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      release: prometheus

serviceMonitorNamespaceSelector is empty, which means Prometheus only monitors the namespace it hosted (monitoring).

serviceMonitorSelector is another important setting that selects ServiceMonitors only with matched labels. Then, let’s check ServiceMonitor kuberhealthy’s configuration:

$ kubectl get servicemonitors kuberhealthy -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
...
  labels:
    app: kuberhealthy
    app.kubernetes.io/managed-by: Helm
    prometheus: prometheus
    release: prometheus-operator
...

Obviously, none of the labels of ServiceMonitor kuberhealthy matches the one listed in the serviceMonitorSelector above.

So the root cause of Kuberhealthy metrics not received by Prometheus is:

• The namespace of ServiceMonitor kuberhealthy is different from Prometheus;
• None of the labels of ServiceMonitor kuberhealthy matches the serviceMonitorSelector of Prometheus.

How to Fix It?

We already knew the root cause, so let’s fix both of them.

Dump the configuration of ServiceMonitor kuberhealthy

$ kubectl get servicemonitors kuberhealthy -o yaml > kuberhealty_servicemonitor.yaml

Update namespace and labels

Open the file of kuberhealty_servicemonitor.yaml in an editor and update the following fields:

• Update the label

• Update the namespace

Apply the configuration file

$ kubectl apply -f kuberhealty_servicemonitor.yaml
servicemonitor.monitoring.coreos.com/kuberhealthy created

List ServiceMonitors again, a kubehealthy ServiceMonitor is shown in monitoring namespace.

$ kubectl get servicemonitors -A
NAMESPACE      NAME           AGE
kuberhealthy   kuberhealthy   76m
monitoring     kuberhealthy   10s
...

Delete the ServiceMonitor kuberhealthy in the namespace of kuberhealthy

Then, the one in the kuberhealthy namespace is useless and can be deleted.

$ kubectl delete servicemonitor kuberhealthy -n kuberhealthy
servicemonitor.monitoring.coreos.com "kuberhealthy" deleted

Verify Again in Prometheus

After the steps above completed and wait for a while, re-check the Service Discovery page in the Prometheus:

The Kuberhealthy metrics are also included in the Graph page:

Once I choose the metric kuberhealthy_check and click the ‘Execute’ button, which displays the records sent from the Kuberhealthy service.

Now, Kuberhealthy is connected with Prometheus!

5. Add Kuberhealthy Dashboard to Grafana

Add Kuberhealthy Dashboard

Kuberhealthy provides a sample Dashboard that displays the status of KHChecks running in your cluster.

https://github.com/Comcast/kuberhealthy/blob/master/deploy/grafana/dashboard.json

Download this JSON file to your local disk and imported it from Grafana by clicking the ‘Upload JSON file’ button.

However, it shows nothing in the imported dashboard:

I tried to edit one chart and found an error message on the editing page:

There must be something wrong with the dashboard definition. Please notice, it says:

Datasource named prometheus was not found

When I check other worked dashboards, the Datasource is Prometheus. What’s the difference?

The first letter is CAPITAL! 😤

Fix Data Source

The fix is easy and straightforward. Replace all prometheus in the data source field Prometheus in the dashboard JSON file.

Delete the previous dashboard in Grafana and import the updated dashboard JSON file.

Bingo!

Now, Kuberhealthy works with Prometheus + Grafana.

Last

Most of the changes in this article are not persistent and only for locally running purposes.

When you redo your Minikube environment, all changes must be done again to make it work.

One approach I recommend is to clone the Kuberhealthy repo to local and make the change directly to your local repo. Next time, install the Kuberhealthy from the local repo instead of the helm, then there is no extra work again.

Now, try and enjoy your local Kuberhealthy!