How to Enable Prometheus on Azure Kubernetes Service
With the growing boom in cloud applications, monitoring, and observability have become essential to ensure the performance and availability of services. Azure Kubernetes Service (AKS) is a leading solution for deploying and managing Kubernetes clusters on Microsoft Azure. Integrating Prometheus as a monitoring solution is a powerful and highly effective option to improve visibility and control over resources in an AKS cluster.
In this article, we’ll explore how to enable Prometheus on Azure Kubernetes, enabling development and operations teams to get accurate and timely metrics about their cloud applications and systems.
Azure Monitor for Prometheus managed service
The Azure Monitor for Prometheus managed service is a feature that allows you to collect and analyze custom metrics, of course, making use of a monitoring solution compatible with Prometheus.
It is essential to know that the Azure Monitor service for Prometheus allows you to use Prometheus query language (PromQL). This allows us to analyze and generate alerts on the performance of all the infrastructure that needs to be monitored and, of course, workloads that need to be monitored.
One thing that attracts the most attention is that it is a component of “Azure Monitor Metrics”. This is very interesting since it gives us flexibility in the types of metric data that you can collect and analyze with the “Azure Monitor” tool.
Data source
An interesting feature is the Azure Monitor service for Prometheus; you can collect data from Azure Kubernetes Service (AKS).
A Data Source for Prometheus in Azure AKS is a location or endpoint from which Prometheus pulls metrics. This data source can be a service or an application running on the AKS cluster that exposes its metrics through an accessible API.
In an AKS environment, containers and applications can expose metrics that describe their performance and internal health. To get these metrics, Prometheus needs to be configured to point to the correct data sources, which are typically the APIs exposed by the services or applications.
The process to enable Prometheus AKS
It must be considered that the only requirement to enable the Azure Monitor service for Prometheus is the creation of an “Azure Monitor Workspace” resource in the Kubernetes cluster, which is where Prometheus will host all the gathered metrics.
Creation Azure Monitor Workspace
Being in the Kubernetes cluster on the left panel, in the “Monitoring” section, select the “Insights” option. Once there, select the “Enable Prometheus” option to enable the wizard for the creation of the “Azure Monitor Workspace”:
In the configuration options, you have to specify the instance of Grafana, which has been previously configured:
Collect Prometheus Metrics From The AKS Cluster
When configuring an AKS cluster to send data to Prometheus in Azure Monitor, a containerized version of the Azure Monitor agent with a metrics extension must be installed. For this, you must specify the “Azure Monitor Workspace” where the data will be sent.
Prerequisites:
- Have an Azure Monitor Workspace configured.
- The cluster needs to use “Managed identity authentication”.
- The following resource providers must be registered in the AKS cluster subscription and Azure Monitor Workspace:
- Microsoft.ContainerService
- Microsoft.Insights
- Microsoft.AlertsManagement
Enable Prometheus Metrics Collection
To install the Azure Monitor agent in the AKS cluster and send the Prometheus metrics to the “Azure Monitor Workspace” you must have an “Azure Monitor Workspace” resource, then select the “Monitored Cluster” option on the left panel to display the list of AKS clusters:
Finally, select the AKS cluster where the metrics collection for Prometheus will be enabled:
By configuring this option, you will allow Prometheus to monitor and record essential data for the performance and health of your applications within the cluster. This provides valuable insights to help you make informed decisions and ensure the optimal operation of your services in Kubernetes.
Verify daemon installation for Azure Monitor agent
The following command checks the ‘DaemonSet’ named ‘ama-metrics-node’ in the ‘kube-system’ namespace to confirm the installation status of the Azure Monitor agent on your Kubernetes cluster:
kubectl get ds ama-metrics-node --namespace=kube-systemNow, verify the presence and status of the replica set within the ‘kube-system’ namespace of your Kubernetes cluster:
kubectl get rs --namespace=kube-systemRemove and stop sending metrics to Prometheus Azure Monitor
First, you need to install the “aks-preview” extension:
az extension add - name aks-previewAfter installing, it’s a good practice to update the extension to make sure you have the latest version:
az extension add - name aks-previewThen we need to remove the agent from the AKS cluster nodes and the rules created for data collection (this does not remove the data already collected) from Azure Monitor:
az aks update --disable-azuremonitormetrics -n <cluster-name> -g <cluster-resource-group>Create and assign permissions for Azure Monitor Workspace
To set permissions in the Azure Monitor Workspace, it’s necessary to register the application in the Microsoft Entra ID.
On the left panel, select the option of “App registrations”. Once there, select the option “New registration”:
Then, enter the name of the application and give it the option to register:
Will see the following window with the new application registration, where we can see the Application (client) ID and Directory (Tenant) ID:
To finish, select in the left panel the option of “Certificates and Secrets”. Select the option “+ New client secret” and enter a label as a description and the expiration period, then click the “Add” button:
Note: The value of the secret can only be seen immediately after creation, so it should be kept in a safe place, as we will need this value later.
Assign permissions to the app for Azure Monitor Workspace
To allow the application to make queries from the “Azure Monitor Workspace” permissions of “Access control (IAM)” must be assigned. On the page “Add role assignment”, in the “Role” tab, look for “Monitoring” select the “Monitoring Data Reader” option, and click on the “Next” button:
Once there, you can select the members by selecting the option “+ Select members” and search for the app created in the previous step of “App registration” created in the Microsoft Entra ID, then click the “Select” button:
Then, click on the “Review + assign” button to assign the role.
Definition of Services and Alert Rules
Prometheus is highly customizable and supports alert rules based on metric queries. Define the services and alerts you want to monitor and receive notifications if certain conditions are met.
Grafana Configuration (Optional)
Although Prometheus provides a basic interface for visualizing metrics, Grafana is a popular tool for creating more sophisticated dashboards and graphs. Configure Grafana to connect to Prometheus and take advantage of its additional features.
Conclusion
Enabling Prometheus on Azure Kubernetes Service offers a robust solution to monitor applications and systems in the cloud effectively. The ability to collect and view real-time metrics enables the development and operations teams to identify potential issues, optimize performance, and ensure an optimal user experience. Combining Prometheus with Grafana provides a powerful and versatile ecosystem for monitoring and analyzing AKS clusters.
