Autoscaling Kubernetes Custom Resource using the HPA

In this article you’ll see how to scale a Custom Resource’s child deployment pods with the Horizontal Pod Autoscaler.

That’s you. You are the captain of your cluster. You shall scale your custom resources with a horizontal pod autoscaler as a captain steers his ship.

If you are building your own Kubernetes Custom Resources (CRs) to manage your applications and want to also have the power to scale those CRs with the Horizontal Pod Autoscaler (HPA) based on the pod metrics of their child deployments, then this post is for you.

I will walk through step by step with code examples to get you set up. I am going to assume you are using either Kubebuilder or Operator-SDK. I am also going to assume you have a custom resource that has a child deployment where you specify the number of replicas for the deployment. Before we begin, make sure your cluster is version 1.11+.

This will be done in approximately 3 steps: updating your CR, updating your controller, and deploying an HPA. At the end of each step we will run a command to verify that it was successful. We’ll start with the CR.

Setting up the Scale Endpoint on your Custom Resource

We’re going to start in our CR types file, in my case: mycr_types.go, by adding some markers (comments that act as metadata and give controller-tools some extra information.

// +kubebuilder:subresource:status
//+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.selector
type MyCR struct {
  //
}

You’ve likely already added the // +kubebuilder:subresource:status line, the important line is the next one where we are defining a scale subresource as well as 3 paths within the scale subresource: .spec.replicas, .status.replicas, and .status.selector. The HPA will use the scale endpoint to retrieve the labelselectors of the child pods, as well as the status and spec of replicas for scaling.

Next we will need to actually add these 3 things to our Status and Spec types, like so:

type MyCRSpec struct {
    Replicas int32    `json:"replicas"
}
type MyCRStatus struct {
    Replicas int32   `json:"replicas"
    Selector string  `json:"selector"`
}

Now that we’ve updated our types, it’s time to modify our CRD and CR. We will start by adding the scale subresource as well as the 3 paths to our CRD:

subresources:
  status: {}
  scale:
    labelSelectorPath: .status.selector
    specReplicasPath: .spec.replicas
    statusReplicasPath: .status.replicas

If you do not already have subresource defined in your CRD, it belongs directly under spec. Notice the status subresource which was also defined in the types file.

Finally, if you haven’t already done so, add a replicas field to your CR:

apiVersion: my.example.com/v1alpha1
kind: MyCR
metadata:
  name: example-cr
spec:
  replicas: 2

Verify (1)

Now let’s verify that the configuration is working as expected. If it is, you should have a scale endpoint on your CR. Apply your CR to the cluster, make sure you have a pod running, and then run the following command: kubectl get --raw /apis/<group_name>/<version>/namespaces/default/<your_cr>/<your_deployed_cr_name>/scale | jq, like so:

$ kubectl apply -f example_cr.yaml
mycr.my.example.com/example-cr created
$ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
example-cr-pod   1/1     Running   0          60s
$ kubectl get --raw /apis/my.example.com/v1alpha1/namespaces/default/mycr/example-cr/scale | jq
{
  "kind": "Scale",
  "apiVersion": "autoscaling/v1",
  "metadata": {
    "name": "example-cr",
    "namespace": "default",
  },
  "spec": {
    "replicas": 1
  },
  "status": {
     "replicas": 0
  }
}

Notice the apiVersion is "autoscaling/v1". As stated earlier, the HPA will be using this endpoint to retrieve the spec and status replicas so it can reconcile them, as well as the label selector (not seen) so it knows which pods to monitor. Also take note that the replicas status is incorrect, we will remedy that next along with the label selector.

Update the CR Status With Deployment Spec

To deal with the missing label selector status, we’re going to need to add a bit of code somewhere in our controller. In our reconcile loop we need to retrieve 2 pieces of information from our child deployment and save them to our CR Spec: the current number of replicas and the pod label selectors (from the deployment spec). Where you put this is up to you, but it will look something like this:

// 1. Retrieve the deployment
deployment := &appsv1beta2.Deployment{}
err = r.client.Get(ctx, types.NamespacedName{Name: name, Namespace: namespace}, deployment)
if err != nil {
  log.Error(err, "Error retrieving Deployment")
  return reconcile.Result{}, err
}
// 2. Retrieve the label selectors from the deployment
import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
selector, err := metav1.LabelSelectorAsSelector(deployment.Spec.Selector)
if err != nil {
  log.Error(err, "Error retrieving Deployment labels")
  return reconcile.Result{}, err
}
// 3. Retrieve the current number of replicas from the 
replicas := *deployment.Spec.Replicas
// 4. Retrieve and update the CR
mycr = &myv1alpha1.MyCr{}
err := r.client.Get(context.TODO(), request.NamespacedName, mycr)
if err != nil {
  log.Error(err, "Error retrieving CR")
  return reconcile.Result{}, err
}
mycr.Status.Selector = selector.String()
mycr.Status.Replicas = replicas
err := r.client.Status().Update(ctx, mlp)

That’s it! For the controller. You will need to import metav1 if you have not already done so to gain access to the LabelSelectorAsSelector method.

Verify (2)

Now rerun your controller/operator and let’s make sure that it is properly updating the selector and replicas status. Rerun the kubectl get — raw command from earlier:

$ kubectl get --raw /apis/my.example.com/v1alpha1/namespaces/default/mycr/example-cr/scale | jq
{
  "kind": "Scale",
  "apiVersion": "autoscaling/v1",
  "metadata": {
    "name": "example-cr",
    "namespace": "default",
  },
  "spec": {
    "replicas": 1
  },
  "status": {
     "replicas": 1
     "selector": "app=example-app"
  }
}

Everything looks good!

Deploy the Horizontal Pod Autoscaler

It’s time for the last step, deploying the HPA. We’re going to set up the HPA to watch our CR and it will pick up the pods from .status.selector. The HPA spec will look something like this:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: example-hpa
  namespace: default
spec:
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - resource:
      name: cpu
      targetAverageUtilization: 2
    type: Resource
  scaleTargetRef:
    apiVersion: my.example.com/v1alpha1
    kind: MyCR
    name: example-cr

We set the targetAverageUtilization really low so we can verify that it is scaling. The important section to look at is scaleTargetRef where we set apiVersion and kind to our CR and name to the name of our deployed CR.

Verify (3)

Now deploy the HPA, type kubectl get hpa -w, lean back in your chair, and watch the magic unfold. After some time, you will hopefully see the status usage go from unknown to known and the scaling start to take effect:

$ kubectl get hpa -w
NAME   REFERENCE      TARGETS     MINPODS   MAXPODS   REPLICAS  AGE
hpa   MyCR/example-cr <unknown>/2%  1       3         0         2s
hpa   MyCR/example-cr 16%/2%        1       3         2         76s
hpa   MyCR/example-cr 9%/2%         1       3         3         2m2s

Additionally, if you examine your CR, you’ll see the HPA has also updated .spec.replicas from 1 to 3.

Summary

To summarize, we began by adding a replicas field to our CR Spec and a selector and replicas field to our CR Status. We then updated our CRD with the scale subresource and these 3 fields for our HPA to reach.

Next, we retrieved the label selector and replicas information from the child deployment and saved it in our CR status. This was done via some logic in our controller’s reconcile loop.

Finally, we deployed our HPA to watch our CR, from which it obtained the child deployment’s pods via the selector. From there, it scales the CR .spec.replicas according to the resource utilization specified. Once it updates the CR .spec.replicas your reconciliation loop will bubble that through to your deployment spec (assuming you had that set up beforehand. If you did not, I will leave that as an exercise but you can follow a very similar protocol using functions that were used in this walkthrough).