A Guide to Service Discovery with Prometheus Operator — How to use Pod Monitor, Service Monitor and Scrape Config.

Prometheus Operator is a popular tool to manage and deploy Prometheus and related monitoring components in Kubernetes. To be able to set up an effective monitoring strategy using it, we need to be familiar with the basics, like how to discover services. Currently this can be accomplished with either Pod Monitor, Service Monitor, and the brand new Scrape Config CRD. In this article we will go over then in a bit more detail.

Disclaimer

You need to have some familiarity with Monitoring concepts, Kubernetes and Prometheus Operator. For introduction to it, please check https://prometheus-operator.dev.

Application running in Kubernetes

To monitor Applications running in Kubernetes, the choice will be between Pod Monitor and Service Monitor, and will depend if you have a Kubernetes Service associated with it.

For example, you might not need create a Kubernetes Service if you are running some sort of Job or Deployment that doesn’t need to handle traffic from or to other applications running inside or outside of your cluster. In that case to discover your Application a simple Pod Monitor will do the job.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: example-app
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: example-app
  podMetricsEndpoints:
  - port: web

Other use cases for Pod Monitor might be Application consists of multiple pods with different metrics or configurations, that cannot be targeted with Service Monitor, like specific need for in configurations like Scrape Intervals, Metric Relabeling, or Metric Endpoint paths that are unique to certain pods within a service.

Service Monitor should be the way to go for the other cases. Either by creating one Service Monitor for each individual Service. Or another strategy might be to use one Service Monitor to target Multiple Applications.

This can be useful, for example, if you want to define some sort of configurations like Metric Relabel Configs or Sample Limits for multiple services in a quick way. You just need to enforce certain values Kubernetes Services, like the same Port Name and Service Labels. Also the Applications targeted by this default Service Monitor need expose metrics under the same metric path. An example below:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: default-service-monitor
  namespace: monitoring
spec:
  endpoints:
    - interval: 10s
      path: /metrics
      port: metrics
      scheme: http
  jobLabel: app.kubernetes.io/name
  namespaceSelector:
    any: true
  sampleLimit: 1000
  selector:
    matchExpressions:
      - key: app.kubernetes.io/name
        operator: Exists

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: my-custom-service
  name: my-custom-service
  namespace: monitoring
spec:
  ports:
    - name: metrics
      port: 9100
      protocol: TCP
      targetPort: 9100
  selector:
    app.kubernetes.io/name: my-custom-service

Outside Kubernetes

If you have your monitoring infrastructure in Kubernetes, but also need to discover services running outside of Kubernetes, for the longest time the only way to do it was via additionalScrapeConfigs. But as a cautionary tale for personal experience, it can become messy very quickly. Just imagine the scenario of a lot of services to monitor and an additionalScrapeConfigs that has more than 2000 lines, and you get the picture — so easy to break with a wrong merge request.

Fortunately the Prometheus Operator community came up with an alternative for the additionalScrapeConfigs just this year, the Scrape Config CRD. It already supports a couple of Service Discovery methods as you can check here. And bellow a quick example how to move services from additionalScrapeConfigs and turn into Scrape Config.

First to deploy the Prometheus stack using the kube-prometheus-stack helm charts, with some values changes like the scrapeConfigSelector to target the Scrape Config we are going to create later on. Also pulling the main image as there were some recent bug fixes at the time of writing this article. Save as main-values and deploy in your namespace.

prometheus:
 prometheusSpec:
   scrapeConfigSelector:
     matchLabels:
       release: prom
prometheusOperator:
 image:
   tag: main

helm upgrade -i prom prometheus-community/kube-prometheus-stack -n monitoring -f main-values.yaml

Then create the Scrape Config files, based on the services you were targeting on additionalScrapeConfigs . Bellow examples for two services, one with Consul SD, and another with Static Config for two databases.

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: mongodb
  labels:
    release: prom
spec:
  consulSDConfigs:
      - server: https://consul-dns:8500
      services:
        -  mongodb
  relabelings:
    - sourceLabels: [__meta_consul_service]
      targetLabel: job

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: cassandra
  labels:
    prometheus: test
    release: prom
spec:
  staticConfigs:
    - labels:
        job: cassandra
      targets:
        - 172.20.0.2:9500

After creating , then access the Prometheus UI and you will be able to check your targets there.

Final Thoughts

Just wanted to give some appreciation for the Prometheus/Prometheus-Operator community, as they were very quick not only to provide the support to Consul on the Scrape Config CRD, just a couple of months after it was first released — which is essential for my current environment setup, that relies heavily on Consul. But also to tackle bug fix as I was doing my first local test with it.

Additionally, I have written about using Service and Pod Monitor with Opentelemetry, you can check it here.

Notes

Since there seems to be a bit confusion about selectors, as per documentation, by default Prometheus Operator defaults to only targeting ServiceMonitor, PodMonitor or ScrapeConfig from it’s own namespace as per documentation. Otherwise you might need to specify in your Prometheus Spec the desired selectors. On the example above for the ScrapeConfig — as we created with the kube-prometheus-stack, is going to then target by default the release name, so the release: prom.