Prometheus federation in Kubernetes
This post will illustrate how we can setup a master/slave federation of Prometheus instances (apparently, we must say Prometheis) in Kubernetes. People who use kubernetes_sd are probably already familiar with the prometheus.io/scrape annotation that can be set on pod specs, as explained here. There is no specific built-in feature in Prometheus for turning on and off scrape on pods. It’s just a usage of the very generic relabeling feature. And we can do something similar for federation.
As explained on Robust Perception, federation is a common way to scale Prometheus, when writing millions of timeseries becomes not enough. This is done with a master Prometheus (or they call it the global Prometheus), which is configured to scrape the slaves. The configuration described here is kind of static: we will have one master and n slaves, n being known. It’s typical for a splitting-by-use strategy: for instance, all the database metrics will be collected by Prometheus A, all the app metrics by Prometheus B, etc. We assume that we can bind every monitored pod to one and only one Prometheus slave.
1. Setup the master
To define the Prometheus master, just create a pod with this specific configuration:
You can see the slave targets listed here. Assuming they’re all in the same namespace, the master will be able to fetch slaves named after their Service name thanks to the Kubernetes DNS service.
Note also the ‘{job=”my-kubernetes-job”}’ match[] param. It tells to federate into master all metrics matching this criteria. In my example, it’s going to be all metrics, but of course in a real-world case I would have to be smarter than that; it should just be a fine-tuned subset of metrics.
2. Setup the slaves
For slaves, the config may look like this:
This is a slightly modified version of the examples found there. Notice how we ignore the relabeling on “__meta_kubernetes_pod_annotation_prometheus_io_scrape”, and declare a relabeling on “__meta_kubernetes_pod_annotation_prometheus_io_slave” instead. It means that every pod with an annotation prometheus.io/slave: slaveA will be handled by this Prometheus instance. The operation can be repeated for every slave wanted, by just replacing slaveA with something else. In OpenShift, this can be advantageously inferred from a template parameter.
Now all that remains is to deploy pods with the annotation:
metadata:
annotations:
prometheus.io/slave: slaveAAnd it will be automatically picked up by the right slave.
I use this kind of federation in discomon: https://github.com/jotak/discomon#demo-4-prometheus-federation-done-with-discomon
