Monitoring Python async gRPC microservices in Kubernetes using Prometheus and Grafana

Monitoring microservices is the cornerstone of ensuring the reliability, performance, and scalability of modern distributed systems. As organizations increasingly adopt microservices architectures to leverage agility and scalability benefits, the need for comprehensive monitoring becomes more pronounced.

In this article, we will see how we can effectively monitor Python async gRPC microservices using Prometheus and Grafana. We will discuss how we can generate metrics for number of requests per minute for our microservices, record the relevant metrics in Prometheus and show them in Grafana.

1. First of all, we will need to integrate a Prometheus interceptor to calculate the required metrics. We used py-grpc-prometheus library and adapted it to work with async gRPC servers: https://github.com/taranisag/py-async-grpc-prometheus
2. Add the interceptor when starting the gRPC server and start an http server which has a metrics endpoint with our microservice’s metrics:

from grpc import aio
from py_async_grpc_prometheus.prometheus_async_server_interceptor import PromAsyncServerInterceptor
from prometheus_client import start_http_server


server = aio.server(interceptors=(PromAsyncServerInterceptor(),))
start_http_server(80)

The interceptor will record the server metrics and the start_http_server function exposes a small http server, which has a metrics endpoint with our microservice’s metrics.

3. We need to scrape the metrics endpoint with a Prometheus instance. For example, in our company we use Google Cloud Managed Service for Prometheus so we created a PodMonitoring resource that scrapes the metrics endpoints and stores the data in the Managed Prometheus instance. A similar approach can be used when using a self hosted version of Prometheus (using the ServiceMonitor or PodMonitor CRD).

4. Finally, we connect the Prometheus instance to Grafana which enables us to visualize and explore metrics such as requests per minute:

Grafana visualization — requests per minute

For example, we used the following PromQL query to get the requests per minute data:

sum(increase(grpc_server_started_total{cluster="...", 
namespace="...", job="..."}[1m]))

Now we can set alerts based on these metrics or even better — scale our microservice (for example, using KEDA’s Prometheus scaler) based on the number of requests per minute.

In summary, in this article we learned how we can monitor our Python async gRPC microservices using Prometheus and Grafana, adjusting existing python gRPC Prometheus metrics library.