Centralise Prometheus metrics using Cortex with an example — Part 2

In the previous blog, we looked at a quick introduction of what Cortex is and some of the advantages it brings along with it.

In this blog, we will look at the implementation of Cortex as a single process i.e.., all the above services will be launched as one service and this is especially suitable in our case as we are just testing things. Apart from the implementation of the demo example, we will look at multi-tenancy in action.

Scenario:

In this demo example, we have the option of installing it with helm or with docker. We will be going ahead with docker as the values in the config are clear and concise as required for initial understanding. After you pull the repo, in the docs/getting-started/, we mainly have the following files:

getting-started
├── docker-compose.yaml
├── cortex-config.yaml
├── grafana-values.yaml
└── prometheus-config.yaml

When we do docker compose up, it creates 4 resources as follows:

$ docker compose up --force-recreate --remove-orphans --detach
[+] Running 4/4
 ✔ Container getting-started-grafana-1     Started                                                                                                                                                                                                                    0.4s 
 ✔ Container getting-started-prometheus-1  Started                                                                                                                                                                                                                    0.1s 
 ✔ Container getting-started-cortex-1      Started                                                                                                                                                                                                                    1.7s 
 ✔ Container getting-started-seaweedfs-1   Started

We are using seaweedfs as a substitute to the S3 bucket and its config is mentioned in the compose yaml file itself. Let’s add a bucket in the seaweedfs, if this is not node, the Cortex container will not work as expected:

$ curl -X PUT http://localhost:8333/cortex-bucket

For viewing the metrics, log into the Grafana instance at http://localhost:3000. Login credentials are username: admin and password: admin and there may be an additional screen on setting a new password. This can be skipped and is optional. Now, add datasource in the following screen:

Select Prometheus in the add Data Source option and in the next screen, add http://cortex:9009/api/prom as the URL value for Prometheus server URL under Connection section. Click on Save & test and we should be able to see a successful popup like this:

We can see metrics in Explore option like this:

With this, we can confirm that the demo setup is working fine. Now, lets see how we can isolate tenants by doing few changes to this example.

Multi-tenancy:

Multi-tenancy in Cortex requires Prometheus to send something called a tenantID along with the metrics and this ID is used to differentiate between tenants. This can be passed using X-Scope-OrgID. But, before that in the cortex-config.yaml, we can see that the auth_enabled option is set to false which basically disables the need/requirement that every request has to send X-Scope-OrgID header. So, we will enable it now as we are looking for multi-tenancy.

In the prometheus-config.yaml, we do the following changes so that the remote_write section looks like this:

remote_write:
  - url: http://cortex:9009/api/v1/push
    headers:
      X-Scope-OrgID: docker-prometheus

Save these files and do docker restart for containers getting-started-prometheus-1 and getting-started-cortex-1 so that the changes are applied.

$ docker restart getting-started-cortex-1
getting-started-cortex-1
$ docker restart getting-started-prometheus-1
getting-started-prometheus-1

Now, in Grafana, the datasource should be throwing the below error:

To fix it, add the following HTTP header and the value is docker-prometheus as we mentioned in the prometheus-config.yaml earlier.

Now, clicking on Save & test should give a success message.

Let us create another Prometheus instance which will act as our second tenant. Copy paste the prometheus-config.yaml and rename it to prometheus-config-2.yaml. Set the X-Scope-OrgID to docker-prometheus-2. The file content will look like this:

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
remote_write:
  - url: http://cortex:9009/api/v1/push
    headers:
      X-Scope-OrgID: docker-prometheus-2

We can add it in docker-compose.yaml file by adding this part:

  prometheus-2:
    image: prom/prometheus:${PROMETHEUS_VERSION}
    command:
      - --config.file=/config/prometheus-config.yaml
    volumes:
      - ./prometheus-config-2.yaml:/config/prometheus-config.yaml:ro
    ports:
      - 9092:9090

We are using port 9092 as 9090 is already used by first Promtheus container. You can bring the second Prometheus container up individually or add the above config to docker compose file and do a docker compose recreate command.

In Grafana, dashboard, even if you refresh, you can’t see the second Prometheus instance because you are isolated from it. In order to view it, we need to add another datasource with HTTP header, X-Scope-OrgID asdocker-prometheus-2. Once added, you will have start seeing a drop-down in the Explore menu as follows:

This shows how X-Scope-OrgID is used and there is another way to pass this ID instead of directly mentioning it under remote_write section in Prometheus yaml files that is, with cortex-tenant.

Cleanup:

$ docker compose down
[+] Running 6/6
 ✔ Container getting-started-grafana-1       Removed                                                                                                                                                                                                                  0.3s 
 ✔ Container getting-started-cortex-1        Removed                                                                                                                                                                                                                  0.0s 
 ✔ Container getting-started-prometheus-1    Removed                                                                                                                                                                                                                 10.7s 
 ✔ Container getting-started-seaweedfs-1     Removed                                                                                                                                                                                                                 10.4s 
 ✔ Container getting-started-prometheus-2-1  Removed                                                                                                                                                                                                                 10.6s 
 ✔ Network getting-started_default           Removed  

That’s it for this blog. Hope you guys have a better understanding of Cortex now.

What’s next?

Cortex-Tenant will be covered in the next blog. We will see on a high level what it brings to the plate and why it is used.