Nerd For Tech
Published in

Nerd For Tech

Deep Dive into Cortex Metrics Part II

Deep Dive into Cortex Metrics.io Part II

In Part I of this article we have understood the various components of Cortex and its use cases. In this Part II, we will configure Cortex with AWS S3 as its backend bucket storage and understand how metrics can be retained for longer periods using Cortex. We will also configure Grafana to use the Cortex Query Frontend to visualize graphs from various Prometheus clusters.

Cortex Metrics

What is the entire story all about? (TLDR)

  1. Install Cortex using the Official helm chart.
  2. Configure Cortex to use AWS S3 as its Object Store.

Prerequisites

  1. A Kubernetes Cluster ( Can be either On-Prem, AKS, EKS, GKE, Kind ).
  2. Helm, kubectl installed.

What is not covered in this article

  1. How to set up a Production-based Cortex cluster.
  2. Compare the various monitoring solutions in the market ( Thanos vs Cortex )

Story Resources

  1. GitHub Link: https://github.com/pavan-kumar-99/medium-manifests
  2. GitHub Branch: cortex

Installing Prometheus Operator.

Well, if you use a package manager like helm, Cortex installation is made easier. Cortex has an official helm chart that can be used to Install a Cortex cluster on a Kubernetes cluster. You can find the helm chart here.

Before we install the Cortex cluster, let us Install the Prometheus Operator using the helm chart here. Later the Prometheus Instance will then be configured with a remote Write configuration to push it to the Cortex Ingesters. But for now, let us install the Prometheus Operator as per your customization. The values for this chart can be configured here. I have however installed consul in a non HA mode. The helm values can be found here.

$ helm repo add prometheus-community \ https://prometheuscommunity.github.io/helm-charts$ helm repo update$ helm upgrade -i prom prometheus-community/kube-prometheus-stack

Installing Cortex in the Kubernetes cluster

The Prometheus Operator is now installed. Let us now install Cortex using the official helm chart. But before that cortex requires the consul to be installed, I have used the official helm chart from the Harshicorp repository. You can follow the steps mentioned in the document here in the cortex namespace.

Consul Helm Chart Installation

As mentioned earlier, for this cortex installation we would be using the AWS S3 bucket as the backend object storage. I have created an AWS secret key and access key from the IAM console and added them to my cortex-values.yaml ( Line 25, 26 ) file and also added the consul URL ( Line 18 ), S3 bucket name ( Line 24 ), and I have also enabled Prometheus Service Monitors with custom labels.

$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b cortex
$ helm repo add cortex-helm \
https://cortexproject.github.io/cortex-helm-chart
$ helm repo update $ helm install cortex --namespace cortex cortex-helm/cortex \
-f cortex-values.yaml
Cortex Installation

Cortex and Consul Components in the cortex namespace

Let us now see, the services present in the cortex namespace. For now, we should consider paying more attention to the following services.

a) cortex-nginx ( http://192.168.29.168 )

b) cortex-query-frontend ( http://192.168.29.171 )

Feel free to pause here and understand the components in Cortex from the Part-I of this series.

Cortex namespaces services

Installing Prometheus Instance using Prometheus Operator ( With RemoteWrite Configuration ).

You may use the Prometheus Custom Resource Definition to create a Prometheus Instance. But the new change that we bring over here is the addition of these 2 lines.

Prometheus Remote Write Configuration

If you notice the URL clearly, you will realize that the URL is the cortex-nginx IP. Now the understanding should be easier. We are sending all the metrics from Prometheus to cortex-nginx which in turn sends to the rest of the cortex components.

$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b cortex
$ kubectl apply -f prometheus-rbac.yaml$ kubectl apply -f prometheus-monitoring.yaml

Now, we should see all the metrics being written to cortex nginx pods, and the metrics are now being processed by the rest of the cortex components. Multiple Prometheus Instances from different namespaces can also push to this cortex instance.

Cortex Nginx receives all the metrics via RemoteWrite

Cortex Front End Querier UI?

But how do I get all of these metrics at a single source? Unfortunately, Cortex doesn't have a Query Frontend UI like Thanos. So we will have to deploy another Prometheus Instance with Remote Read Configuration. This configuration will read from the Cortex Query Frontend EndPoint. If you now see line 23, you should realize the fact that the IP is the IP of cortex-query-frontend.

Cortex Remote Read Configuration
$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b cortex
$ kubectl apply -f prometheus-remote-read.yaml

You should now see the metrics from all the various Prometheus clusters being visible here. Isn’t that cool? Yes, it is !!

Metrics from various Prometheus Clusters

And now the time comes to verify the main thing. Yeah, the retention. I have been running cortex metrics in my home cluster for quite a long time. I now wanted to get the compute resources utilization for the past 30 days. Can I do it? Yes, obviously because I have already configured cortex to use AWS S3 bucket as its backend block storage. Ahh, here is the Data.

Cortex Performance Dashboard:

There’s a beautiful dashboard here, to visualize the performance of Cortex in Grafana. I have tuned it by changing a couple of panels and queries and making it look more attractive. I have added that to my repo, feel free to suggest some modifications to the Dashboard.

Cortex Performace metrics

Conclusion

Thus, with the help of Cortex and Prometheus, one can set up a HA metrics aggregation solution. While there are many other solutions available in the market, the cortex is one of the solutions which is used by many companies in their Production environment. Initially, I had a hard time understanding every component of Cortex. But exploring each of the components individually has given me greater insights into Cortex Metrics. Feel free to share your experiences with Cortex in the comments section below.

Until next time…..

Recommended

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Pavan Kumar

Pavan Kumar

586 Followers

Cloud DevOps Engineer at Informatica || CKA | CKS | CSA | CRO | AWS | ISTIO | AZURE | GCP | DEVOPS Linkedin:https://www.linkedin.com/in/pavankumar1999/