[DIY] How to Set Up Prometheus and Ingress on Kubernetes

A simple guide to get started with Prometheus.

Ankit Deshpande
Gojek Product + Tech
7 min readMay 12, 2020

--

In the previous post we discussed about the TIG stack and how we can set it up on Kubernetes.

Today let’s take a look at how we can setup Prometheus and use it to monitor Kubernetes cluster and other resources running on the cluster. We will also look in into how to setup NGINX Ingress Controller and monitor it using Prometheus.

Let’s start with some introductions

Prometheus
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud.

It records real-time metrics in a time series database built using a HTTP pull model, with flexible queries and real-time alerting. It’s written in Go, and its choice of query language is PromQL.

Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.

Ingress
Ingress allows HTTP and HTTPS traffic from outside the cluster to services within the cluster. Traffic is routed based on the rules defined on the Ingress resource.

An Ingress may be configured to give services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting.

Ingress can be shared, which means it can be used to route traffic among multiple services running on the cluster, and same AWS/ GCP load balancer can be used. No more creating load balancer per service, This also helps in keeping the AWS / GCP cost down through better resource utilisation.

Note: This is not a production ready configuration/setup, it is for experimenting with how Prometheus, Nginx Ingress work and how to set them up.

Prerequisites

We will set up the following components:

  • Prometheus
  • Dashboard on Grafana to monitor K8 Cluster Metrics
  • Ingress using Nginx Controller
  • Dashboard on Grafana to monitor Nginx Ingress

Set up Prometheus

For Prometheus, we will

  • Install Prometheus
  • Configure a dashboard on Grafana

We can install a lean Prometheus setup using helm charts:

Lets dry-run the helm chart to verify what components it will create (a good practice to follow before deploying any changes to production cluster)

If no repos are found:

Now to create the resources, lets remove the dry-run flag:

The following deployments will be created by the chart:

Deployments created by the helm chart.

Following DaemonSet will be created by the chart:

DaemonSet created by helm chart.

To Access Grafana

Let’s head to Grafana (http://192.168.99.100:31246) and configure Prometheus as a Data source.

Grafana > Configurations > Add Data sources > Select Prometheus

Configure Prometheus server URL with the kubernetes service endpoint

Setup Prometheus data source in Grafana.

Dashboard for viewing Kubernetes Cluster Metrics

There are a couple of standard dashboards already available on https://grafana.com/grafana/dashboards. Creating customised dashboards on Grafana can be covered in another post.

I have used this one: https://grafana.com/grafana/dashboards/10856, you can choose another one depending on your preference. 😃

To Import Dashboard

K8 Dashboard listing all the deployment workload running on the cluster.

Setup Ingress

For Ingress, we will

  • Create a sample service/workload
  • Install Nginx Controller
  • Create Ingress Resource
  • Dashboard on Grafana

Create a sample workload

Install Nginx-Controller

On Minikube:

On K8 Cluster Using Helm:

Create Ingress Resource

Create ing.yaml with similar resource definition, point it to service we created above:

Run the command to create Ingress resource:

To access our sample workload using Ingress:

To Create Dashboard

After you have configured the dashboard, we can generate some sample load by making some requests using CURL

We can see metrics being reported on the dashboard:

Nginx Ingress metrics like CPU, Memory Usage etc
Percentile Response Times and Transfer Rates, Certificate Expiry Details

Wait, what, how did it start reporting metrics ?

We didn’t configure any targets for Prometheus to scrape/pull the metrics from, so how did it start working ?

Let’s describe NGINX Pod

Description of the NGINX Pod

If we hit, http://172.17.0.2:10254/metrics on the K8 Cluster, it returns a list of metrics which the monitoring module on NGINX has captured:

Now, let’s look at Prometheus and try to figure out how it is pulling these metrics from the NGINX Pods.

To Access Prometheus Dashboard

Prometheus Targets

Let check http://localhost:9090/targets, this will list down all the targets that Prometheus will scrape metrics from. Targets can be configured statically using static_configs or dynamically using one of the supported service-discovery mechanisms.

Let’s look at the NGINX Target:

NGINX Pod target on Prometheus

Prometheus Service Discovery

Check out NGINX Service Discovery

http://localhost:9090/service-discovery

NGINX Pod Discovered

Discovered Pod

There are a couple of other discovered pods as well, for now we are only interested in NGINX Pods.

Conclusion

This is just a setup guide to begin playing around with NGINX Ingress and Prometheus.

Prometheus also has a lot of other components:

  • Prometheus Server
  • Alert Manager
  • Push Gateway
  • Exporters

Prometheus can be used for a lot of other things. It can scrape custom application metrics as well, all the application has to do is to implement a /metrics or a similar endpoint and expose metrics that are to be monitored.

Using PromQL, it can create alerts based on thresholds for metrics.

Using AlertManager, it can decide which channels/integrations to deliver alerts to.

We will take a look at Prometheus in depth some other time.

A few helpful commands

Useful Links
Prometheus:
https://prometheus.io/

Prometheus Configuration: https://prometheus.io/docs/prometheus/latest/configuration/configuration/

Grafana: https://grafana.com/grafana/

Ingress: https://kubernetes.io/docs/concepts/services-networking/ingress/

Deploy Nginx Controller: https://kubernetes.github.io/ingress-nginx/deploy/

Node Exporter: https://github.com/prometheus/node_exporter

Cortex Project: Horizontally scalable, highly available, multi-tenant, long term Prometheus.
https://github.com/cortexproject/cortex
https://cortexmetrics.io/

You can also find the yaml resource definitions here.

That’s all from me for now. Hope you found this post useful!

Liked what you read? We can have our stories delivered straight to your inbox. Sign up for our newsletter!

--

--