Effortless Microservice Monitoring: Mastering Dynamic Probing with Prometheus & Blackbox in Kubernetes

If you’re managing expansive workloads on Kubernetes, or dealing with a rapidly growing number of microservices, this simple yet valuable monitoring hack could be your secret weapon!

If one or more of the microservices hosted in the Kubernetes infrastructure goes down, the entire workload may be affected, and if the problems are not detected within the specified SLA period, we may encounter unexpected and unfortunate consequences. As you know, automated monitoring solutions are always our savior in order to avoid these consequences.

As it is known, there are open source or paid tools for probing endpoints that can work independently or integrated with each other. In this article, we will talk about how we can dynamically probe service endpoints using Prometheus & Blackbox. (We will proceed assuming that you have already installed the relevant tools in your Kubernetes workload.)

A rapidly growing number of microservices generally means an increase in the number of service objects we host on Kubernetes and, accordingly, the service endpoints that need to be probed regularly.

Generally, service endpoints or URLs to be probed are probed by adding them as targets to a ‘blackbox job’ in prometheus.yml. You can see an example of this configuration below:

- job_name: 'blackbox-example'
  scrape_timeout: 15s
  scrape_interval: 15s
  metrics_path: /probe
  params:
    module: [http_2xx] #default blackbox HTTP module that could be found on blackbox.yaml
  static_configs:
   - targets:
      - https://example.com #URL
      - my-application.prod:80 #a service endpoint located on 'prod' namespace
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115 #blackbox exporter service endpoint

Our focus will be on probing HTTP service endpoints.

You can manually add each newly deployed service endpoint as a target as seen in the example. There is absolutely no harm in doing so, but it is obvious that it will make your job difficult.

Instead, you can have blackbox dynamically discover and probe each newly added service endpoint, as seen in the example below!

- job_name: 'blackbox'
  scrape_interval: 15s
  scrape_timeout: 15s
  metrics_path: /probe
  kubernetes_sd_configs:
  - role: service #specifies which object type will be probed
    namespaces:
      names:
        - prod #specifies the namespace of these objects
  params:
    module: [http_2xx]
  relabel_configs:
    - source_labels: [__address__]
      regex: '^(.*):?(\d+)?$' #refers to the service endpoint format
      replacement: '$1/actuator/health' #adds /actuator/health path to the end of objects that fit the format
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - target_label: __address__
      replacement: prometheus-blackbox-exporter:9115

You may prefer to automate this process for a specific namespace, a set of namespaces, or for every namespace in your workload!

Looking at the difference, we can see that the main part that provides the dynamism is the following part:

- role: service #specifies which object type will be probed
  namespaces:
    names:
      - prod #specifies the namespace of these objects

You can also change the role definition with different objects such as ‘pod’, but please keep in mind that in this case, the regex and replacement logics specified later in the job configuration may also need to be changed or removed.

The main purpose of making these monitoring configurations is that we want to view the up/down status of microservices through their healthchecks.

Now, let’s look into the other differences that made under the relabel_configs part.

#first example:
- source_labels: [__address__] #retrieves the value of the current __address__ tag
  target_label: __param_target #places it in the target tag __param_target

#second example:
- source_labels: [__address__] #retrieves the value of the current __address__ tag
  regex: '^(.*):?(\d+)?$' #checks whether the value of the __address__ tag follows a certain format
  replacement: '$1/actuator/health' #adds /actuator/health path to the end of objects that fit the format
  target_label: __param_target #places the final value in the target tag __param_target

You can consider this configuration as a magic because it adds the healthcheck path to the end of the specified service endpoint format!

In this example, a format in which you can probe backend microservices written in Java & Spring Boot has shown, but of course you can revise the healthcheck path, make additions and deletions, taking into account the healthcheck configurations of your microservices in your architecture.

Explanation of the regex format mentioned in the example:

^(.*):?(\d+)?$ -> <service-name>.<namespace>:<port>

This is an example format, you can change it as well according to your architectural decisions. Keep in mind that, if there are microservices with different healthchecks that you probe with the same blackbox job, you will definitely need to make adjustments on this healthcheck configuration.

After these changes are made in prometheus.yml, you can view them from the Targets tab under the Status menu from the Prometheus UI. In the Targets tab, you will be able to view a job named “blackbox” (may vary depending on the value you assign to the job_name variable in the yml file) and the service endpoints tracked within this job.

States of the service endpoints should be “UP” which means that Blackbox is able to probe them. But remember that their status being UP does not indicate that these endpoints respond with the required HTTP code.