Smart observability practices brings infrastructure cost-optimization to the table

Are you interested in automating rightsizing capabilities either your team or entire organisation? I bet you do!

Let’s dive right in and see how pushing boundaries of your rightsizing initiative may look like.

Observability in Cloud Native environment

Vertical Pod recommender

Recommender is a service that provides recommendations and insights for using resources in your Kubernetes environment.

To evolve rightsizing in cloud native environment with relatively little invest effort you can bring a good portion of value to the table, meaning it is definitely a valuable and impactful initiative.

In the end of the story almost any team member can confidently start rightsizing user workloads they have ownership within Kubernetes cluster and start saving money to the company or at least have control over optimal infrastructure resources utilization. Chances are, business stakeholders may buy-in your FinOps initiative.

First things first, I assume you have already adopted VPA autoscaller and walked through installation bits including setup recommender moving part.

Next thing would be to take care of configuration bit on kube-state-metrics side. These steps are required to expose VPA recommendations.

Foremost, allow kube-state-metrics object to talk to relevant Kubernetes APIs

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
...
rules:
  - apiGroups:
      - apiextensions.k8s.io
      - autoscaling.k8s.io
    resources:
      - customresourcedefinitions
      - verticalpodautoscalers
    verbs:
      - list
      - watch

Last but not least, define your custom resources and the fields to turn into metrics

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
spec:
  template:
    spec:
      containers:
        ...
        args:
          - --custom-resource-state-config
          -  |
              spec:
                resources:
                  - groupVersionKind:
                      group: autoscaling.k8s.io
                      kind: "VerticalPodAutoscaler"
                      version: "v1"
                    labelsFromPath:
                      verticalpodautoscaler: [metadata, name]
                      namespace: [metadata, namespace]
                      target_api_version: [apiVersion]
                      target_kind: [spec, targetRef, kind]
                      target_name: [spec, targetRef, name]
                    metrics:
                      - name: "vpa_container_recommendations_target"
                        help: "VPA container memory recommendations."
                        each:
                          type: Gauge
                          gauge:
                            path: [status, recommendation, containerRecommendations, "0", target, memory]
                        labelsFromPath:
                          container: [status, recommendation, containerRecommendations, "0", containerName]
                        commonLabels:
                          resource: "memory"
                          unit: "byte"
                      - name: "vpa_container_recommendations_target"
                        help: "VPA container cpu recommendations."
                        each:
                          type: Gauge
                          gauge:
                            path: [status, recommendation, containerRecommendations, "0", target, cpu]
                        labelsFromPath:
                          container: [status, recommendation, containerRecommendations, "0", containerName]
                        commonLabels:
                          resource: "cpu"
                          unit: "core"

Now then, since we do have custom recommendations metrics available at /metrics endpoint, we are ready to move on to the next moving part, the visualization layer.

# HELP kube_customresource_vpa_container_memory_recommendations_target VPA container recommendations for memory.
# TYPE kube_customresource_vpa_container_memory_recommendations_target gauge
kube_customresource_vpa_container_memory_recommendations_target{container="name",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace",resource="memory",target_api_version="autoscaling.k8s.io/v1",target_kind="Deployment",target_name="target-name",unit="byte",verticalpodautoscaler="vpa-name"} 2.62144e+08
# HELP kube_customresource_vpa_container_cpu_recommendations_target VPA container recommendations for cpu.
# TYPE kube_customresource_vpa_container_cpu_recommendations_target gauge
kube_customresource_vpa_container_cpu_recommendations_target{container="name",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace",resource="cpu",target_api_version="autoscaling.k8s.io/v1",target_kind="Deployment",target_name="target-name",unit="core",verticalpodautoscaler="vpa-name"} 0.025

I’ll share how to plot a relevant graph in upcoming article. Talk to you later!

Direct instrumentation with Prometheus ruby client