Monitoring Optimimization of K8s microservices with Prometheus and Stackdriver

This article is written by Hadrien Leonard, DevOps intern at vpTech.

Introduction

During my third year at Epitech, I enrolled in a four months full-time internship at Veepee. I joined the Customer Experience Optimisation team as a DevOps developer to work on monitoring, alerting and especially on a stability issue.

We are using two Prometheus instances. One shared across the whole data department and one dedicated to the team. Currently the shared instance often experiences down times, it impacts directly the team monitoring system, leading to potential information loss.

The objective has been to migrate our team metrics to Stackdriver, to keep us independent from the global instance and make our monitoring system more reliable.

In the following article, you will discover CXO’s monitoring infrastructure and how we try to improve it.

Current monitoring infrastructure

Currently, the monitoring schema is as follow:

Processes are dockerized, and managed under kubernetes clusters. Their metrics are exposed and scraped by Prometheus instances. Then, Grafana is plugged into Prometheus in order to create dashboards, thanks to the collected metrics.

The main goal is to migrate from Prometheus, to Google Cloud Monitoring (Stackdriver), to retrieve everything into Google and to make the monitoring infrastructure more reliable. We don’t absolutely want to exclude Prometheus from the stack, but add Stackdriver to store the metrics. Before diving deeper into the implemented solution, we will first see different possible solutions, to weight up pros and cons.

Monitoring re-architecturing

First, we started to explore a solution excluding the usage of Prometheus, by sending the metrics directly to Google Cloud Monitoring. We started to work on Apache Airflow migration using their documentation. The solution would have been to use statsd, an applicative library that exposes metrics, to deal with metrics export, and then implement an exporter in order to collect them.

This would have turned out time consuming and we would have to implement a unique solution for each process we had to migrate.

The solution we proposed was to keep using Prometheus, but only as a scraper. As a result, metrics would still be collected by Prometheus, but directly sent to Google Cloud Monitoring thanks to a sidecar container. Because Prometheus is widely used for our deployments, this solution is by far the easiest to implement, as we only have to modify the Prometheus deployment.

Here is the part of the deployment file handling the containers. The first container (line 1 to 19) is the Prometheus backend, scraping all the deployment metrics endpoint. Then, a second container, the sidecar, has been added (line 20 to 43). It shares the same storage volume with the Prometheus backend (line 39–40), in order to get direct access to collected metrics, to push them to Google Cloud Monitoring (Stackdriver).

We use Skaffold and Helm to manage the deployment process. Thus, we have a file containing variables concerning the configuration. We use official images stored in a private repo for security purposes.

However, you can find the official Google documentation, in which you will find the hard coded values, such as the container image.

As shown above, you only need to add the sidecar container to your prometheus deployment.. You must

set configuration arguments in the “args” field.

- “ — stackdriver.project-id”={your_project_id}

- “ — prometheus.wal-directory=”/{storage_folder}/wal

- “ — stackdriver.kubernetes.location”={cluster_zone}

- “ — stackdriver.kubernetes.cluster-name”={cluster_id}

As a reminder, “storage_folder” is a shared volume with prometheus. As a result, you need to mount the same volume on the two containers, in order to share the data collected by prometheus with the sidecar. We experienced several issues while setting up the sidecar container. To get debug messages in the container logs, you can add “ — log-level=debug”. You will get much more verbose logs, which is very convenient to troubleshoot the deployment.

As you may notice, it is much easier and faster than editing every single deployment to implement it’s own metrics exporting method. All metrics are centralized in the storage volume, thanks to prometheus, then the sidecar only has to push them to Google Cloud Monitoring, through Stackdriver api.

The sidecar needs permissions to call Google’s API.
Don’t forget to grant the deployment a service account with the role “roles/metricsWriter’, to grant it access to the api.
We use environment variables and secrets to mount the Google credentials json file.
First, you need to create a service account for the sidecar container.
We use Terraform to provision resources on GCP. Here is the code snippet to provision the required service account, with the needed role line 11:

Then, you need to create a secret, to store the JSON key, created by GCP.

This secret stores the base 64 encoded key, given from a template variable, into a secret.

Following step is to mount the secret onto the container, in order to give it the credentials key.

Once everything is done, you can add the GOOGLE_APPLICATION_CREDENTIALS environment variable, containing a path to the credentials.

Currently, we are facing different challenges. The deployment works, sidecar is running, but it meets several errors. We have a lot of metrics to export, with tons of labels. However, Stackdriver Api is limited to 10 labels per metrics, see quotas and limits. This is a hard limitation you cannot bypass.

“level=debug ts=2021–07–08T13:12:08.013Z caller=series_cache.go:380 component=”Prometheus reader” msg=”too many labels” labels=”{__name__=\”istio_tcp_connections_closed_total\”,connection_security_policy=\”unknown\”,destination_app=\”unknown\”,destination_canonical_revision=\”latest\”,destination_canonical_service=\”unknown\”,destination_principal=\”unknown\”,destination_service=\”kubernetes.default.svc.cluster.local\”,destination_service_name=\”kubernetes\”,destination_service_namespace=\”unknown\”,destination_version=\”unknown\”,destination_workload=\”unknown\”,destination_workload_namespace=\”unknown\”,instance=\”10.44.46.63:15020\”,job=\”kubernetes-pods-containers\”,k8s_namespace=\”drk-test\”,k8s_pod_name=\”agora-slot-var2-generator-schedulermax1625718600x699n\”,reporter=\”source\”,request_protocol=\”tcp\”,response_flags=\”-\”,source_app=\”unknown\”,source_canonical_revision=\”latest\”,source_canonical_service=\”agora-slotvar2-generator-scheduler-max\”,source_principal=\”unknown\”,source_version=\”unknown\”,source_workload=\”agora-”slot-var2-generator-scheduler-max\”,source_workload_namespace=\”drk-test\”}””

As shown above, we can observe istio metrics getting blocked because of too many labels in the metric. We have tons of messages like this for each different istio metric.

We have investigated different solutions. The first one would be to exclude labels from metrics. However, it’s very time-consuming, probably several days just to write the exclusion rules, for each deployment. In addition, there would be information loss.
Nevertheless, we are not sure that all metrics are concerned by the limitations issues, may be only istio.

level=debug ts=2021–07–08T13:12:08.008Z caller=client.go:173 component=storage msg=”Partial failure calling CreateTimeSeries” err=”rpc error: code = Internal desc = Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs."

Thus, as suggested by this error message, there is probably an error in the configuration, which leads to strange behavior. In addition, it looks more logical because messages concerning labels excess only concern Istio. We do not have any messages for any other deployment, which is weird because all the metrics are stored in the shared volume. Effectively the previous messages let us know it has access to the metrics, even if the sidecar cannot use them.
Thus, we are investigating the container deployment manifest of the sidecar, in order to find what’s wrong.

Conclusion

As a conclusion, there are several ways to monitor your processes. You can use specific methods for each of them, or use one to centralize all metrics from every deployment. We decided to use Prometheus, and plug an exporter on the top of it, in order to make maintenance easier through all deployments.

Keep in mind that Prometheus and Google Cloud Monitoring are the most common solutions for metrics storage on GCP. Google Cloud Monitoring is a fully managed Google service, which is supposed to be handier, because you only have to configure the client side, without caring about any instances. On the contrary, Prometheus is a third-party monitoring solution. You must configure both client and server side. Indeed, you must deploy a Prometheus backend, and configure it to scrap the metrics from your application, which must also be configured to expose metrics. Nevertheless, metrics exposing on the client side is very easy to set-up.

In our case, we decided to use both because Prometheus is very convenient as it is already fully configured, while Stackdriver can offer more than monitoring, thanks to other Google services, such as built-in alerting.
Thus we benefit from Prometheus ease for scraping the metrics, then thanks to the sidecar exporter, we are also able to push the metrics over Google Api, to retrieve them from GC monitoring.

During my internship, I enjoyed my experience. The working environment is ideal to learn and there is a strong team spirit allowing easy communication through teammates. This 4 months experience allowed me to discover all the fields related to data, such as data scientist and data engineer. This is a domain that interests me enormously and in which I intend to evolve.

Last but not least, I would like to thank Vivian OLIVERES , Guillaume CHAUSSERON and more generally the whole team. Thanks to them, I learnt a lot as a DevOps intern, I really feel comfortable with the whole stack. It was a real opportunity to work together.