Requests & Limits recommendations using VPA, Goldilocks and Grafana

When you develop a cloud native application on Kubernetes, you always need to customize and optimize CPU and RAM usage of your deployments.

If you don’t do it can lead to unpleasant situations, like not enough resources to manage a node loss, or some OOMKilled process due to some too restrictive memory limits. Moreover, default values for some deployments (like some public helm charts) does not correspond to your workload.

So, how can you check your deployments, and adapt requests and limits to match as best as you can? We can do it using a mix of open-source tools, let’s have a look!

Vertical Pod Autoscaler

Kubernetes Vertical Pod Autoscaler can track CPU and RAM usage, and adapt automatically the pod requests and limits based on metrics. You can find more details here.

VPA has 3 components: Recommender, Updater and Admission Plugin. To have recommendations only, you will only deploy Recommender.

Installation is done using a helm chart provided by Fairwinds, and VPA can also get Prometheus metrics, so let’s install and activate it:

$ helm repo add fairwinds-stable https://charts.fairwinds.com/stable
$ helm repo update
$ helm install vpa fairwinds-stable/vpa --namespace vpa --create-namespace -f vpa-custom-values.yaml # See below

VPA Helm Custom Values

Here, I use Prometheus as a metric provider, to have more accurate values. So you need to install it (it’s not a pre-requisite, but you shall have a monitoring system for you cluster).

Now that you have VPA up and running, you can create a VPA configuration. Check an example here. That’s pretty simple, but I want more automation here, and here comes Goldilocks!

Goldilocks

Goldilocks is a utility from Fairwinds to identify requests and limits, based on the Kubernetes VPA, as just as you installed.

It tracks deployments, and automatically creates VPA based on labels! So, if you want to create VPA, just add labels, and goldilocks does the rest.

Installation is done using the provided helm chart:

$ # Goldilocks Helm repository is already there
$ helm install goldilocks --namespace vpa fairwinds-stable/goldilocks

Once goldilocks is deployed, you have 3 pods running:

• Goldilocks controller, which tracks labels and creates VPA accordingly
• Goldilocks dashboard, which adds some dashboard to see VPA recommandations
• VPA recommender, which calculates CPU and limits recommendations

$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
goldilocks-controller-5bf99   1/1     Running   0          49d
goldilocks-dashboard-699f4    1/1     Running   0          49d
vpa-recommender-74fdc         1/1     Running   0          67d

Now, just add labels to some namespace and see the magic : I add it on the veleronamespace, and vpa is automatically created:

$ kubectl label ns velero goldilocks.fairwinds.com/enabled=true
$ kubectl get namespace velero --show-labels                  
NAME     STATUS   AGE    LABELS
velero   Active   112d   goldilocks.fairwinds.com/enabled=true,name=velero
$ kubectl get vpa -n velero                 
NAME                MODE   CPU   MEM         PROVIDED   AGE
goldilocks-restic   Off    15m   183046954   True       67d
goldilocks-velero   Off    11m   183046954   True       67d

Note that Fairwinds also provides a chart to install goldilocks and vpa at the same time, but they not recommend it.

Great, you have now VPAs automatically created, and we can also see recommendations on a basic dashboard!

Goldilocks Dashboard

OK, that’s fine, but I have already some nice dashboards with my monitoring system using Prometheus and Grafana, why not use it instead of the Goldilocks one? Let’s go!

Prometheus & Grafana

Digging into VPA settings, you can see that kube-state-metrics exposes some metrics like:

• kube_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound
• kube_verticalpodautoscaler_status_recommendation_containerrecommendations_upperbound

All available metrics are defined here. If you are using the prometheus operator stack, you just have to configure the verticalpodautoscalers collector in the kube-state-metrics deployment and it will scrape theses values.

Prometheus VPA Metrics

You have now VPA, Prometheus metrics, you just need to stop goldilocks dashboard and create our own in Grafana, getting accurate metrics!

1. Update the goldilocks helm chart to disable dashboard

$ helm upgrade goldilocks --namespace vpa fairwinds-stable/goldilocks -f goldilocks-custom-values.yaml

Goldilocks Custom Values

2. Create a Grafana dashboard with all these metrics!

Grafana VPA Recommendations

Nice! You can track your Requests and Limits from your preferred dashboard, and adapt your workloads to fit your cluster size. You can also see recommendations history, to fine tune your deployments.

You can download the Grafana dashboard here: https://grafana.com/grafana/dashboards/16294

Fun fact, I used it to adapt goldilocks requests and limits ;)

Resources

You can get one predefined Grafana dashboard here, which I used as inspiration for mine.