Calming down Kubernetes Autoscaler

Kubernetes Autoscaler is a huge money saver! It constantly monitors cluster and tries, if possible, to re-distribute pods to maximize utilization of nodes. Unfortunately sometimes it’s just too aggressive, but with one small trick Kubernetes Autoscaler can be calmed down!

For example, Cirrus CI can run CI tasks on Google Kubernetes Engine clusters. Cirrus CI uses Kubernetes Jobs API to schedule a job for each CI task. When we just came up with this approach and were load testing it, it appeared, that Kubernetes Autoscaler was killing active jobs and re-starting them on other nodes to maximize overall utilization. In general, it makes sense for long-running jobs and pods, but in our case we know for sure that the CI jobs has timeouts and usually finishes just in a few minutes. There is simply no reason to restart a CI job on another node.

After digging into internals of Kubernetes Autoscaler, it appeared that by default it doesn’t evict pods with local storages. So the eviction can be avoided simply by mounting a emptyDir!

apiVersion: v1
kind: Pod
metadata:
  name: test-pd
spec:
  containers:
  - image: k8s.gcr.io/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /tmp
      name: temp-volume
  volumes:
  - name: temp-volume
    emptyDir: {}

That’s all! Now Kubernetes Autoscaler won’t touch your pods! But in case you still want to make exceptions for some pods, you can simply add cluster-autoscaler.kubernetes.io/safe-to-evict: true annotation to such pods.

Update (09/28/2018): cluster-autoscaler.kubernetes.io/safe-to-evict: false annotation works now.

Please read FAQ on Kubernetes Autoscaler for more in-depth details. I hope this little blog post will help to avoid unexpected evictions in your clusters!