Enabling ArgoCD metrics and Monitoring Using Kube-Prometheus-Stack

6 min readOct 2, 2023

What is ArgoCD and Kube-Prometheus-Stack?

ArgoCD is an open-source, declarative, GitOps continuous delivery tool for Kubernetes. Its primary purpose is to automate the deployment and management of applications in Kubernetes clusters. ArgoCD uses Git repositories as the source of truth for declarative application configurations and provides a user-friendly interface for administrators and developers to define, deploy, and synchronize applications.

The kube-prometheus stack is a collection of open-source monitoring and alerting tools and configurations designed to provide comprehensive observability for Kubernetes clusters. It’s built on top of Prometheus, a popular open-source monitoring and alerting toolkit. The kube-prometheus stack includes various components and configurations that enable monitoring and alerting for Kubernetes and its workloads. Key components include: Prometheus, Grafana, AlertManager, Prometheus Operator and Exporters (Node Exporter and Kube-state-metrics)

How are ArgoCD metrics exposed and what can be monitored?

Monitoring the components of an ArgoCD deployment can include the Application Controller, ArgoCD Server, and Repo Server. They can be monitored on different ports allowing you to gain insights into the health, performance, and operational status of these components.

Application Controller

The Application Controller in ArgoCD is responsible for managing and reconciling the desired state of applications. When monitoring the Application Controller, you can collect and analyze the following metrics and information:

Application Sync Status: You can monitor the synchronization status of applications, including whether they are up-to-date or have encountered synchronization errors.
Resource Health: Monitor the health of resources (e.g., pods, services) within applications to detect issues such as crashes or excessive resource usage.
Application Deployment History: Track the history of application deployments and rollbacks to identify trends and anomalies.
Performance Metrics: Gather performance metrics related to the Application Controller’s resource consumption and response times.
Error and Event Logs: Collect logs and events generated by the Application Controller to diagnose and troubleshoot issues.

ArgoCD Server

The ArgoCD Server is the core component that provides the web-based user interface and API for managing applications and GitOps workflows. When monitoring the ArgoCD Server, you can capture the following information:

API Request Metrics: Monitor the usage and performance of ArgoCD’s REST API, including the number of requests, response times, and error rates.
User Authentication and Authorization: Track user logins, authentication failures, and access control events to ensure security and compliance.
Resource Usage: Monitor server resource utilization (CPU, memory) to detect performance bottlenecks and scaling needs.
Application Activity: Gather information about application CRUD (Create, Read, Update, Delete) operations and deployments.
Dashboard Metrics: If ArgoCD provides a web-based dashboard, you can monitor dashboard usage, user interactions, and navigation patterns.

Repo Server

The Repo Server in ArgoCD is responsible for interacting with Git repositories to fetch application manifests and sync application states. When monitoring the Repo Server, you can collect the following data:

Git Repository Sync: Monitor the status of Git repository synchronization, including sync errors and repository fetch times.
Resource Caching: Track the caching of Git repository resources, which can help optimize performance by reducing the need for frequent fetches.
Authentication and Access: Monitor Git repository authentication and access control to ensure secure access to Git repositories.
Resource Integrity: Verify the integrity of Git repository data and identify any corruption or inconsistency issues.
Resource Usage: Monitor resource utilization of the Repo Server, such as CPU and memory consumption.

Let’s get to Work!

In this article, I will only be guiding how to expose metrics via Application Controller and ArgoCD-Server and collecting it in prometheus end using its CRDs.

ArgoCD-Server Metrics (8083)

Edit the argocd-server svc file (kubectl edit svc argocd-server -n argocd)
If not present in argocd-server svc file, add metrics endpoint port with target as 8083 under spec.ports

Edit the argocd-server deployment file (kubectl edit deployment argocd-server -n argocd)
In the argocd-server deployment file, under spec.template.annotations add the following:

In the argocd-server deployment file, under spec.template.spec.containers.args add the following:

Also under ports.containerPort -> change the port to 8083 where needed
After all the above steps are done, you are able to verify if the metrics are being exposed at the 8083 port.
To do that: kubectl port-forward -n argocd svc/argocd-server 8083
Check at localhost:8083/metrics to see whether the metrics are available.

Upon successful verification, a servicemonitor can be set up in prometheus end in order to look for the metrics at this endpoint.
ServiceMonitor for argocd-server metrics:

SVC name should be specified precisely i.e: argocd-server

Upon successful configuration of the ServiceMonitor, it should be visible in the prometheus UI in this manner.

Application Controller Metrics (8082)

Argocd-application-controller is a statefulset in this case. We can check the statefulset configuration by doing a: kubectl edit statefulset <name> -n <namespace>

Under spec.template.metadata.annotations add:

A point worth noting is to ensure that under matchLabels in the Service monitor or Pod monitor, the label should be accurately specified if not it could lead to a misconfiguration of the CRDs.
Infact, the name specified should match with the key-value pairs in the relevant pod or service under the selectors. It can be seen in the below attached image.

Upon the succession of these steps, we can run the following command to port-forward where the metrics can be collected at the 8082 port: kubectl port-forward -n argocd pod/argocd-application-controller-0 8082
Similar to the argocd-server, it can be verified whether the metrics are exposed from argocd by checking the localhost:8082/metrics endpoint

Getting the metrics to prometheus

A PodMonitor had to be created as the application-controller metrics are exposed via a statefulSet. If a servicemonitor was created for this purpose, the argocd-application-controller would not be listed as one of the active targets, therefore labels would be dropped and hence the label mentioned under matchLabels in the serviceMonitor would not be matched. Therefore, a podmonitor has to be implemented.