Shutting down Kubernetes dev environments with KEDA

The last months we’ve been working on our new container platform based on Kubernetes and implementing a GitOps way of working with ArgoCD. We’re already using AWS a lot and we decided to go with EKS for our Kubernetes clusters.

When we sat down to think about auto-scaling pods we decided we want Prometheus to play a part in this. Within Wehkamp we use it a lot for our monitoring and reusing that data for scaling would be nice. Enter KEDA, it is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed.

While testing KEDA we experimented with some options and one option is to pause auto-scaling. But not only pause it but also set the number of pods while paused. Combined with the fact that KEDA, unlike HPA, can scale to zero, we immediately thought about our test and development environments. What better way to save costs than to scale pods to zero! Karpenter will destroy the worker nodes when there are no more pods running.

We have a very basic implementation running at the moment and only time will tell if that is good enough when the clusters keep getting bigger. But for now it is perfect for us.

The technical details:

To pause the scaling and set the number of pods to zero, we have to annotate the scaledObject. We can do that with a simple one-liner:

kubectl annotate scaledObject -n namespace servicename “autoscaling.keda.sh/paused-replicas=0”

And to reverse it and resume auto-scaling:

kubectl annotate scaledObject -n namespace servicename autoscaling.keda.sh/paused-replicas-

If we put this in a cronjob we can schedule the stop and start of the auto-scaling pause. And of course we don’t want manual actions, so we just need to dynamically create a list with all the scaledObjects present.

Example of the stopjob with some basic scripting with kubectl to get all the scaledobjects, parse it with jq and put it in a bash array to annotate all the scaledobjects we found:

In the evening we run this job. In the morning we run the startenv cronjob that is almost identical, but removes the annotation, so auto-scaling is enabled again and pods will start. This means that outside our office hours there are no pods running in our non-production environments (except our core services like external-dns, karpenter and such). That way we can save some $$$.

If you’re interested in the complete setup of these cronjobs and how to get it as an ApplicationSet in ArgoCD, let me know!

Note, that’s over here now :

https://medium.com/wehkamp-techblog/argocd-applicationset-for-cronjob-5d1ed552a303