Image for post
Image for post
Photo by AltumCode on Unsplash

CODEX

Setup Kuberhealthy with Prometheus and Grafana on Minikube

Rocky Chen
Jan 10 · 9 min read

I browsed lots of online resources and instructions to setup Kuberhealthy with Prometheus and Grafana together on a local Kubernetes environment like Minikube, but unfortunately, none of them works for me.

Most of the instructions and articles listed installation steps for each of them but seldom showed the steps for:

  • How to connect Kuberhealthy and Prometheus? That means how to configure Kuberhealthy service to send metrics to Prometheus service successfully.
  • How to configure Grafana dashboard to display the Kuberhealthy metrics collected by Prometheus.

There are some extra steps needed for most cases after you installed them and this post would show you how to make them work together.

All of these tools are running on macOS (10.15.7).

1. Minikube Setup

To install Minikue on Mac is easy by using brew:

brew install minikube

Once Minikube gets installed successfully, you could start a local Kubernetes cluster quickly with it.

I created a local cluster with two nodes with the following command:

$ minikube start -n 2
😄 minikube v1.16.0 on Darwin 10.15.7
✨ Automatically selected the docker driver. Other choices: hyperkit, virtualbox
👍 Starting control plane node minikube in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=4050MB) ...
🐳 Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: storage-provisioner, default-storageclass
👍 Starting node minikube-m02 in cluster minikube
🤷 docker "minikube-m02" container is missing, will recreate.
🔥 Creating docker container (CPUs=2, Memory=4050MB) ...
🌐 Found network options:
▪ NO_PROXY=192.168.49.2
🐳 Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
▪ env NO_PROXY=192.168.49.2
🔎 Verifying Kubernetes components...
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

From the output, we could learn the version for Minikube v1.16.0 and Kubernetes v1.20.0.

Then, run a kubectl command to verify it looks good:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 3m11s v1.20.0
minikube-m02 Ready <none> 2m9s v1.20.0

2. Prometheus + Grafana Setup

I choose the kube-prometheus-stack because it includes both Prometheus Operator and Grafana dashboards so I don’t need to set up them separately.

kube-prometheus stack, a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

Let’s install it in a separate namespace monitoring for kube-prometheus-stak with helm.

$ kubectl create namespace monitoring
$ kubectl config set-context --current --namespace=monitoring

Before installation, make sure you’ve added the repo to helm:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install prometheus prometheus-community/kube-prometheus-stack

Check if the Prometheus and Grafana installed correctly:

$ kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 2m49s
prometheus-grafana ClusterIP 10.99.119.38 <none> 80/TCP 2m50s
prometheus-kube-prometheus-alertmanager ClusterIP 10.102.243.73 <none> 9093/TCP 2m50s
prometheus-kube-prometheus-operator ClusterIP 10.106.211.104 <none> 443/TCP 2m50s
prometheus-kube-prometheus-prometheus ClusterIP 10.101.155.118 <none> 9090/TCP 2m51s
prometheus-kube-state-metrics ClusterIP 10.110.98.157 <none> 8080/TCP 2m50s
prometheus-operated ClusterIP None <none> 9090/TCP 2m48s
prometheus-prometheus-node-exporter ClusterIP 10.98.183.105 <none> 9100/TCP 2m50s

To verify Prometheus working as expected, we could expose its webpage by minikube service command below:

$ minikube service prometheus-kube-prometheus-prometheus -n monitoring

The webpage is opened in your default browser like:

Image for post
Image for post
The default webpage of Prometheus

Let’s access Grafana by exposing its service with the Minikube command:

$ minikube service prometheus-grafana -n monitoring

You would see the login page of Grafana opened in your default browser and fill in the credential of username: admin and password: prom-operator.

Image for post
Image for post
The default webpage of Grafana

When going to the Dashboard->Manage page, there is a list of the pre-set dashboard for you.

Image for post
Image for post
Image for post
Image for post
The list of pre-set dashboards in Grafana.

This is another reason for me to choose kube-prometheus-stack to install.

3. Kuberhealthy Setup

Once Prometheus and Grafana are ready in the Minikube cluster, Kuberhealthy can be installed via helm as well.

Typically, Kuberhealthy is installed in the kuberhealthy namespace.

$ kubectl create ns kuberhealthy
$ kubectl config set-context --current --namespace=kuberhealthy

Since kube-prometheus-stack uses Prometheus Operator, Kuberhealthy should also be installed for Prometheus Operator. It would install a Kuberhealthy ServiceMonitor which can be discovered by Prometheus Operator, and then Kuberhealthy is possible to send the metrics to Prometheus.

$ helm repo add kuberhealthy https://comcast.github.io/kuberhealthy/helm-repos$ helm repo update$ helm install kuberhealthy kuberhealthy/kuberhealthy --set prometheus.enabled=true,prometheus.enableAlerting=true,prometheus.serviceMonitor=true
  • First, there should be a kuberhealthy service in the kuberhealthy namespace.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kuberhealthy ClusterIP 10.102.231.49 <none> 80/TCP 91s
  • There should 3 khchecks installed by default in the kuberhealthy namespace. khcheck is a custom resource created by Kuberhealthy in the cluster to run tests. For details, please read here.
$ kubectl get khchecks
NAME AGE
daemonset 3m5s
deployment 3m5s
dns-status-internal 3m5s
  • The most important, ensure kuberhealthy checks able to send results to kuberhealthy service. It could be done by checking the Pod status. The pods related to khchecks should be Completed status.
+ kubectl get pods
NAME READY STATUS RESTARTS AGE
check-reaper-1610257860-778w5 0/1 Completed 0 6m22s
check-reaper-1610258040-5vjjs 0/1 Completed 0 3m21s
check-reaper-1610258220-mg4lb 0/1 Completed 0 21s
daemonset-1610257533 0/1 Completed 0 11m
deployment-1610257533 0/1 Completed 0 11m
deployment-1610258132 0/1 Completed 0 109s
dns-status-internal-1610257652 0/1 Completed 0 9m49s
dns-status-internal-1610257772 0/1 Completed 0 7m49s
dns-status-internal-1610257892 0/1 Completed 0 5m49s
dns-status-internal-1610258012 0/1 Completed 0 3m49s
dns-status-internal-1610258132 0/1 Completed 0 109s
kuberhealthy-7b5b54ddb7-p7c7s 1/1 Running 0 12m
kuberhealthy-7b5b54ddb7-xwj9j 1/1 Running 0 12m

If the status of Pods is not Completed, please refer to one of my previous posts: Minikube NATs Source IP Address of Pods for services with ClusterIP

4. Connect Kuberhealthy to Prometheus

Most instructions end by step 3 after installing all the components, but actually, they are not working together yet. Because all the settings and configurations are in default values which might not match each other.

Let’s check if Prometheus can receive Kuberhealthy metrics first.

  • Goto Service Discovery page in Prometheus to check if Kuberhealthy is on the list. By default, Kuberhealthy service is not listed.
Image for post
Image for post
  • On the Graph page, check if Kuberhealthy metrics included. In my example, they are not included.
Image for post
Image for post

Therefore, Prometheus is unable to receive Kuberhealthy metrics yet, and of course, Grafana doesn’t have Kuberhealthy data either.

First, let’s have a look at ServiceMonitors in all namespaces because Prometheus Operation collecting metrics via them.

$ kubectl get servicemonitors -A
NAMESPACE NAME AGE
kuberhealthy kuberhealthy 29m
monitoring prometheus-kube-prometheus-alertmanager 3h35m
monitoring prometheus-kube-prometheus-apiserver 3h35m
monitoring prometheus-kube-prometheus-coredns 3h35m
monitoring prometheus-kube-prometheus-grafana 3h35m
monitoring prometheus-kube-prometheus-kube-controller-manager 3h35m
monitoring prometheus-kube-prometheus-kube-etcd 3h35m
monitoring prometheus-kube-prometheus-kube-proxy 3h35m
monitoring prometheus-kube-prometheus-kube-scheduler 3h35m
monitoring prometheus-kube-prometheus-kube-state-metrics 3h35m
monitoring prometheus-kube-prometheus-kubelet 3h35m
monitoring prometheus-kube-prometheus-node-exporter 3h35m
monitoring prometheus-kube-prometheus-operator 3h35m
monitoring prometheus-kube-prometheus-prometheus 3h35m

The ServiceMonitor of Kuberhealthy is the top of the list which means Kuberhealthy follows Prometheus’ pattern to provide a ServiceMonitor to send metrics.

But please notice that it is in the kuberhealthy namespace instead of monitoring namespace.

Another check is for Prometheus. Prometheus here is another custom resource definition (CRD) created by Prometheus Operator.

$ kubectl get Prometheus -A
NAMESPACE NAME VERSION REPLICAS AGE
monitoring prometheus-kube-prometheus-prometheus v2.22.1 1 3h41m

We could display its configuration via kubectl command. Here is a part of its configuration in YAML:

$ kubectl get Prometheus prometheus-kube-prometheus-prometheus -n monitoring -o yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
name: prometheus-kube-prometheus-prometheus
namespace: monitoring
...
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
matchLabels:
release: prometheus

serviceMonitorNamespaceSelector is empty, which means Prometheus only monitors the namespace it hosted (monitoring).

serviceMonitorSelector is another important setting that selects ServiceMonitors only with matched labels. Then, let’s check ServiceMonitor kuberhealthy’s configuration:

$ kubectl get servicemonitors kuberhealthy -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
...
labels:
app: kuberhealthy
app.kubernetes.io/managed-by: Helm
prometheus: prometheus
release: prometheus-operator
...

Obviously, none of the labels of ServiceMonitor kuberhealthy matches the one listed in the serviceMonitorSelector above.

So the root cause of Kuberhealthy metrics not received by Prometheus is:

  • The namespace of ServiceMonitor kuberhealthy is different from Prometheus;
  • None of the labels of ServiceMonitor kuberhealthy matches the serviceMonitorSelector of Prometheus.

We already knew the root cause, so let’s fix both of them.

Dump the configuration of ServiceMonitor kuberhealthy

$ kubectl get servicemonitors kuberhealthy -o yaml > kuberhealty_servicemonitor.yaml

Update namespace and labels

Open the file of kuberhealty_servicemonitor.yaml in an editor and update the following fields:

  • Update the label
Image for post
Image for post
  • Update the namespace
Image for post
Image for post

Apply the configuration file

$ kubectl apply -f kuberhealty_servicemonitor.yaml
servicemonitor.monitoring.coreos.com/kuberhealthy created

List ServiceMonitors again, a kubehealthy ServiceMonitor is shown in monitoring namespace.

$ kubectl get servicemonitors -A
NAMESPACE NAME AGE
kuberhealthy kuberhealthy 76m
monitoring kuberhealthy 10s
...

Delete the ServiceMonitor kuberhealthy in the namespace of kuberhealthy

Then, the one in the kuberhealthy namespace is useless and can be deleted.

$ kubectl delete servicemonitor kuberhealthy -n kuberhealthy
servicemonitor.monitoring.coreos.com "kuberhealthy" deleted

Verify Again in Prometheus

After the steps above completed and wait for a while, re-check the Service Discovery page in the Prometheus:

Image for post
Image for post
Kuberhealthy shown in the Service Discovery list.

The Kuberhealthy metrics are also included in the Graph page:

Image for post
Image for post
The Kuberhealthy metrics can be selected.

Once I choose the metric kuberhealthy_check and click the ‘Execute’ button, which displays the records sent from the Kuberhealthy service.

Image for post
Image for post
Metrics records for kuberhealthy_check.

Now, Kuberhealthy is connected with Prometheus!

5. Add Kuberhealthy Dashboard to Grafana

Kuberhealthy provides a sample Dashboard that displays the status of KHChecks running in your cluster.

https://github.com/Comcast/kuberhealthy/blob/master/deploy/grafana/dashboard.json

Download this JSON file to your local disk and imported it from Grafana by clicking the ‘Upload JSON file’ button.

Image for post
Image for post

However, it shows nothing in the imported dashboard:

Image for post
Image for post

I tried to edit one chart and found an error message on the editing page:

Image for post
Image for post

There must be something wrong with the dashboard definition. Please notice, it says:

Datasource named prometheus was not found

When I check other worked dashboards, the Datasource is Prometheus. What’s the difference?

The first letter is CAPITAL! 😤

The fix is easy and straightforward. Replace all prometheus in the data source field Prometheus in the dashboard JSON file.

Image for post
Image for post

Delete the previous dashboard in Grafana and import the updated dashboard JSON file.

Image for post
Image for post

Bingo!

Now, Kuberhealthy works with Prometheus + Grafana.

Last

Most of the changes in this article are not persistent and only for locally running purposes.

When you redo your Minikube environment, all changes must be done again to make it work.

One approach I recommend is to clone the Kuberhealthy repo to local and make the change directly to your local repo. Next time, install the Kuberhealthy from the local repo instead of the helm, then there is no extra work again.

Now, try and enjoy your local Kuberhealthy!

CodeX

Everything connected with Code & Tech!

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store