Monitor Kubernets Services/Endpoints with Prometheus Blackbox Exporter

Happy devSecOps

6 min readDec 3, 2022

Background

In modern scalable systems designed as microservices-based distributed systems architecture. Typically these microservices deployed using Kubernetes-based container orchestration systems. The services in the systems interacts with different external and internal endpoints. For an example external endpoints would be third party web Services/APIs etc which the services interacts with. The internal endpoints would be the different services/endpoints inside the system which deployed with Kubernetes. Monitoring these different endpoints is an important feature for diagnosing performance and availability issues. For an example we may need to monitor the availability of external APIs and if one of these API goes down we’d want to be notified immediately. In the current context, endpoint monitoring refers to monitoring internal and external endpoints (HTTP/S, DNS, TCP, and ICMP) for various parameters including HTTP/TCP latencies, HTTP/TCP endpoint availability, DNS lookup latencies, SSL certificate expiry information, TLS version etc.

In this post I’m gonna discuss about monitoring different endpoints/services in a Kubernetes cluster using Prometheus Blackbox Exporter. All the deployments which related to this post available in gitlab. Please clone the repo and continue the post.

Prometheus Balckbox Exporter

Prometheus is an open-source, metrics-based monitoring system. Prometheus does one thing, and it does it well. It has a powerful data model and a query language to analyze how applications and infrastructure perform. The Prometheus stack is composed of multiple pieces, the Prometheus server that stores and serves the data, the Alert Manager that manages the alerts, and tons of Prometheus exporters that perform the metric collection. An exporter is software dedicated to one thing, fetching statistics from another application and exposing them to a specific endpoint (generally, a port and a path) to allow the collection of those metrics by a remote Prometheus server. Exporters fetch statistics from a non-prometheus system and can convert the statistics to Prometheus understandable metrics, i.e. the Prometheus exposition format.

Blackbox is one of the official exporters maintained by the Prometheus organization. The Blackbox exporter is a tool that allows engineers to perform one simple thing that every system administrator does every day, check the availability of HTTP/S, DNS, TCP, and ICMP endpoints. Basically, the Prometheus Blackbox exporter can be seen as a free simple alternative to PingDOM, Datadog, Freshping, or Uptime.com to monitor internal endpoints not exposed on the Internet. After you define the endpoint, Blackbox Exporter generates metrics that can be visualized using tools like Grafana. One of the most important feature of Blackbox Exporter is measuring the response time of endpoints. The following diagram shows the flow of Blackbox Exporter monitoring an endpoint.

Following are the main steps I have followed to monitoring the endpoint(services, endpoints, pods etc)of Kubernets cluster using Blackbox Exporter. I have installed the Prometheus and Blackbox Exporter using the Helm charts.

Install/Configure Blackbox Exporter

I have used prometheus-community/prometheus-blackbox-exporter Helm chart to install the Blackbox Exporter. Following is the way to install and configure Blackbox Exporter with Helm chart. Blackbox Exporter uses different modules(HTTP, TCP etc) to probe different endpoints. In here I have used TCP-based module(named tcp_prober) since I’m gonna probe the Kubernetes services/endpoints. I have defined this TCP module configuration in a blackbox-values.yaml file and inject it to Blackbox Exporter Helm chart.

Install/Configure Prometheus

Once installed the Blackbox Exporter, I can install Prometheus and configure it to probe the targets with previously defined Blackbox Exporter tcp_prober module. Following is the way to installed Prometheus with prometheus-community/kube-prometheus-stack Helm chart. I have defined prometheus configurations in the prometheus-values.yaml file. The serviceMonitorNamespaceSelector: {} flag instructs to select ServiceMonitors on all namespaces, serviceMonitorSelector: {} instructs to get all existing ServiceMonitors.

Monitor Kubernets Services

In this scenario I’m gonna monitor Kubernets services via Blackbox Exporter. To demonstrate how Blackbox Exporter monitoring works, I have run few Kubernets services with pods. Main thing to notice here is, I have added a label app.kubernetes.io/component: blackbox to the each service. In the monitoring config(e.g ServiceMonitor with tcp_prober), I have specified only to monitor the services with the label.

Blackbox Exporter implements the multi-target exporter pattern where we can pass multiple targets to probe. When probing, the Blackbox Exporter needs to be passed the target as a parameter, this can be done with relabelling in Prometheus scrape config. Following is the Prometheus ServiceMonitor I have used to probe Kubernetes services in the cluster. The ServiceMonitor uses the tcp_prober module defined in the Blackbox Exporter. It scrapes Kubernets services via tcp_prober module of the Blackbox Exporter and publish the matrixes under /probe path.

When running this ServiceMonitor it will starts to probe the Kubernets services with app.kubernetes.io/component: blackbox label in all namespaces. If want, we can specify the namespace to monitor via namespaceSelector in the ServiceMonitor. Once run the ServiceMonitor, Prometheus starts to probe the metrics of different Kubernets services via the tcp_prober module defined in the Blackbox Exporter. The collected metrics and the configurations/targets related to tcp_prober can be found in the Prometheus UI. I have port-forward the prom-kube-prometheus-stack-prometheus service and accessed the Prometheus UI through localhost.

The configurations of the Prometheus with tcp_prober can be found in Status tab -> Configurations. We can check whether the Blackbox Exporter is up with the registered targets by navigating to the Status tab -> Targets in the Prometheus UI. We can also check if metrics are getting populated by looking for metrics starting with probe_(e.g probe_success displays whether or not the probe was a success).

Monitor Kubernets Endpoints

Now I’m gonna monitor Kubernets endpoints using Blackbox Exporter. We can create a custom endpoint that will be pointed to any desired resource(e.g Pod, Service, External database etc). For the demonstration purpose, I have created ClusterIP: None services(without selector) and point created endpoints which points to the services.

Then I have created ServiceMonitor to probe the endpoints. The ServiceMonitor uses the tcp_prober module defined in the Blackbox Exporter. It scrapes Kubernets endpoints via tcp_prober module of the Blackbox Exporter and publish the matrixes under /probe path. When running this ServiceMonitor it will starts to probe the Kubernets services with app.kubernetes.io/component: blackbox label in all namespaces. The collected metrics and the configurations/targets related to tcp_prober can be found in the Prometheus UI. I have port-forward the prom-kube-prometheus-stack-prometheus service and accessed the Prometheus UI through localhost.

Similar to service monitoring scenario, the configurations of the Prometheus with tcp_prober can be found in Status tab -> Configurations. We can check whether the Blackbox Exporter is up with the registered targets by navigating to the Status tab -> Targets in the Prometheus UI. We can also check if metrics are getting populated by looking for metrics starting with probe_(e.g probe_success displays whether or not the probe was a success).