Deploying Prometheus and Grafana monitoring stack to Kubernetes the GitOps way using ArgoCD

Introduction to Prometheus, Grafana, and ArgoCD

Prometheus is a monitoring solution for recording and processing any purely numeric time series. It gathers, organizes and stores metrics along with unique identifiers and timestamps.

Prometheus is open-source software that collects metrics from targets by “scraping” metrics HTTP endpoints. Supported “targets” include infrastructure platforms (e.g. Kubernetes), applications, and services (e.g. database management systems). Together with its companion AlertManager service, Prometheus is a flexible metrics collection and alerting tool.

Grafana: Grafana is open-source software that enables you to query, visualize, alert, and explore your metrics, logs, and traces wherever they are stored.

As a visualization tool, Grafana is a popular component in monitoring stacks, often used in combination with time series databases such as InfluxDB, Prometheus, and Graphite; monitoring platforms such as Sensu, Icinga, Checkmk, Zabbix, Netdata, and PRTG; SIEMs such as Elasticsearch and Splunk; and other data sources.

A Kubernetes cluster dashboard sample on Grafana

ArgoCD: ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. ArgoCD is implemented as a Kubernetes controller which continuously monitors running applications and compares the current, live state against the desired target state. A deployed application whose live state deviates from the target state is considered OutOfSync.

Helm Charts: Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Helm is a DevOps tool that streamlines installing and managing Kubernetes applications. The tool packages configuration files in a format called charts.

When you look at the Diagram above, your thought process should be whenever there is an Application or a bunch of Applications you need to be deployed with complete flexibility and manageability over them, Helm Charts are the best suggestion for this.

Prerequisites

• ArgoCD installed in a running cluster(if you have not done this then please refer to this documentation to set it up.
• A GitHub repository connected to the ArgoCD Application

Setting up your manifests

Create 2 files in the folder of the Repository you have connected to ArgoCD and customize the following values using Prometheus and Grafana’s global helm charts

• Prometheus.yaml

# Description: Prometheus application for ArgoCD############################################
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prometheus
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: monitoring
    server: https://kubernetes.default.svc
  project: default
  sources:
    - repoURL: https://github.com/<git-username>/<git-repo>.git
      targetRevision: HEAD
      ref: <customizable reference>
    - repoURL: https://prometheus-community.github.io/helm-charts/
      chart: prometheus
      targetRevision: 19.7.2
      helm:
        values: |
            server:
              service:
                type: LoadBalancer
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
  
  
# ############################## END OF prometheus.yaml ####################################

the above yaml file creates the Prometheus application from the Prometheus helm chart and exposes it as a LoadBalancer along with other customizable fields, refer here if you need more values.

• grafana.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: grafana
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: monitoring
    server: https://kubernetes.default.svc
  project: default
  sources:
  - repoURL:  https://github.com/<git-username>/<git-repo>.git
    targetRevision: HEAD
    ref: <customizable reference>
  - repoURL: https://grafana.github.io/helm-charts
    chart: grafana
    targetRevision: 6.52.2
    helm:
      values: |
        service:
          type: LoadBalancer
        datasources:
          datasources.yaml:
            apiVersion: 1
            datasources:
              - name: Prometheus
                type: prometheus
                url: http://prometheus-server.monitoring.svc.cluster.local
                access: proxy
                isDefault: true
              - name: loki
                type: loki
                url: http://loki.monitoring.svc.cluster.local
                access: proxy
        dashboardProviders:
          dashboardproviders.yaml:
            apiVersion: 1
            providers:
              - name: "default"
                orgId: 1
                folder: ""
                type: file
                disableDeletion: false
                editable: true
                options:
                  path: /var/lib/grafana/dashboards/default
        dashboards:
          default:
            kubernetes:
              gnetId: 10000
              revision: 1
              datasource: Prometheus
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

the above yaml file creates the Grafana Application from the Grafana helm chart. This one does more than just create it and expose it as a LoadBalancer though, After exposing it as a load balancer it adds two data sources which are Loki and Prometheus, adds a dashboard provider then adds a dashboard called dashboard 10000. Most of these things would have been done manually but with this, it can be easily declared here.

Now that we have successfully set up our manifest and made sure we have the Repository folder with these manifests connected to our ArgoCD Application, the next step would be to check if they have been deployed successfully on the Application, if they are not we only need to refresh or sync the the base app on ArgoCD that they are in and they should reflect.

If the Prometheus PVC is having issues with being ready you need to add the EBS CSI Driver to your cluster add-ons and set the required permissions, then sync again.

Also, If you are having Challenges creating a Base App on ArgoCD you can follow these steps:

Step 1: Log in to Argo CD

Step 2: Connect your GitHub repo to Argo CD ( the Argo CD within the cluster ), navigate to settings, and select a+ connect repo

Step 3: create a new app with the repo URL that will contain your yml/yaml file.

Step 4: make sure your App looks like this

Conclusion

In conclusion, deploying the Prometheus and Grafana monitoring stack to Kubernetes the GitOps way using ArgoCD provides numerous benefits and streamlines the entire monitoring process. By leveraging the power of GitOps, organizations can achieve enhanced observability, scalability, and reliability in their Kubernetes infrastructure.

The adoption of Prometheus allows for robust metrics collection, monitoring, and alerting, enabling proactive identification and resolution of issues. With Grafana’s intuitive dashboards and visualization capabilities, teams can gain valuable insights into their system’s performance and make informed decisions to optimize resource allocation and improve overall efficiency.

ArgoCD serves as a powerful tool for GitOps-driven deployments, ensuring consistent and auditable application rollouts across Kubernetes clusters. Its declarative approach enables version-controlled management of configurations and promotes collaboration between development and operations teams. With ArgoCD, organizations can automate the deployment process, reduce manual errors, and maintain a reliable and scalable monitoring stack.

By combining these technologies, organizations can establish a comprehensive monitoring solution that aligns with modern DevOps practices. The GitOps approach, empowered by ArgoCD, brings increased transparency, traceability, and reproducibility to the deployment process while ensuring the monitoring stack remains in sync with the desired state defined in the Git repository.

In summary, deploying Prometheus and Grafana monitoring stack to Kubernetes using ArgoCD presents a robust and efficient approach to monitoring containerized environments. It empowers teams to effectively manage their infrastructure, gain actionable insights, and make informed decisions, ultimately contributing to the overall success and stability of their applications.