Deploying Prometheus Multi-Cluster monitoring using Prometheus Agent Mode

In the previous post I wrote about Prometheus Multi-cluster monitoring and how using Prometheus in agent mode helps create a single pane of glass for monitoring multi Kubernetes clusters. Therefore, if you haven’t read it yet, please read it before this hands-on post. In this post, we are going to deploy Prometheus agent mode along with a Prometheus global view and test how these work together in action. For this tutorial, you need a Kubernetes Cluster and two separate namespaces, monitoring-global, and monitoring. Are you ready to run?!

Photo by Andrea Piacquadio from Pexels

Deploy the Global view Prometheus

First make sure that you have already created both needed namespaces. otherwise, create them with the following command:

kubectl create ns monitoring-global && kubectl create ns monitoring

All the files I am using to deploy on the test cluster are available to download or clone on my Github.

1- Deploy a config map to be used in the Prometheus deployment. This config map creates prometheus.rules which defines rule statements and prometheus.yml which is the configuration file.

kubectl apply -f prometheus-global-view/config-map-global.yml

2- Deploy the Prometheus deployment. Since it is only for tests and not that much data is going to be stored here, we add an emptyDirVolume and also we enable the remote-write-receiver which allows Prometheus to accept remote write requests from other Prometheus servers.

kubectl apply -f prometheus-global-view/prometheus-global-deployment.yml

If you get the running pods, you see the created pod.

$kubectl get pods -n monitoring-global                                                                                                                                                                        
NAME                                     READY   STATUS    RESTARTS   AGE
prometheus-deployment-6d84cb9b8b-5r2zb   1/1     Running   0          2m35s

3- It is better to create a headless service to use as the remote-write endpoint in the agent-mode namespace by the agent-mode Prometheus.

kubectl apply -f prometheus-global-view/headless-service.yml

Now that headless-service is created, you can simply forward a local port to it and access the running Prometheus global view.

$kubectl port-forward svc/prometheus-global-headless-service 9090:9090 -n monitoring-global                                                                                                                
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

Call the health check endpoint to make sure that everything is working. Or you can browse http://localhost:9090 on your browser.

$curl http://localhost:9090/-/healthy                                                                                   
Prometheus Server is Healthy.

Deploy the Agent-mode Prometheus

Now it is time to deploy Prometheus in the agent mode and remote-write the metrics to the global-view one.

1- Create a ClusterRole and a ClusterRoleBinding for Prometheus to be able to scrape some Kubernetes metrics.

kubectl apply -f prometheus-agent/clusterrole.yml

2- We need to create a config map to be used as the agent-mode Prometheus server configuration file. Here we are adding the remote_write endpoint.

kubectl apply -f prometheus-agent/config-map.yml

3- Now we deployed all the prerequisites for the agent-mode deployment, finally, we can deploy the agent mode Prometheus. We are enabling the agent mode installation by passing the --enable-feature=agent argument.

kubectl apply -f prometheus-agent/prometheus-deployment.yml

By deploying the last piece, now the Prometheus agent mode is running and you can confirm it by forwarding its pod port to your localhost.

$kubectl get pods -n monitoring                                                                                                                                                                                           
NAME                                    READY   STATUS    RESTARTS   AGE
prometheus-deployment-fd7f6557c-tvsjj   1/1     Running   0          
$kubectl port-forward prometheus-deployment-fd7f6557c-tvsjj 9080:9090 -n monitoring                                                                                                                                        
Forwarding from 127.0.0.1:9080 -> 9090
Forwarding from [::1]:9080 -> 9090

And now if you browse localhost:9080 on the browser you get the following page which shows that the Prometheus is running in the agent mode.

And now, if you port forward the global view again and query for the available metrics, you will see metrics shipped from the agent one accessible on it.

Conclusion

Prometheus in agent mode is useful when you want to monitor multiple clusters in a single pane of glass. But it is not only the case, these days many companies are moving forward to implement edge computing. It is the era of IoT, self-driving cars, and many other models that you can deploy a Kubernetes cluster in a resource-bounded device.