Enabling HPA Important Warning

Enabling Kubernetes Horizontal Pod Autoscaling (HPA) Important Warning

If you have an existing deployment where HPA is currently disabled and you later enable it, replicas will reset to 1 until the HPA controller kicks in.

This process only takes a few moments, but if your containers take several minutes to start, it can cause impact to your application.

Future deployments after HPA is enabled will not touch the replica count, only the first time HPA is enabled will the number of replicas reset to 1, brand new deployments with HPA enabled will start at 1 replica before the HPA controller adjusts the number of replicas.

Why the number of replicas is reset to 1 ?

At he first time HPA is enabled and by removing the replicas from your configuration, k8s will check the last-applied configuration and it finds the last configured replicas, but your new deployment with HPA enabled doesn’t contains the replicas property, k8s will reset to 1 pod instead of 0 before the HPA controller adjusts the number of replicas.

How to fix that

It is possible to work around this behavior if you wish to enable HPA and cannot tolerate the brief reset to 1 replica, but it is a manual process by following those steps :

Step 1 : View if the last-applied exists

kubectl apply view-last-applied deployment/your-service

Step 2 : If the last-applied exists then go to step 5, otherwise continue the steps

• Download your deployment configuration

kubectl get deploy your-service -o yaml > my-service.yml

Step 3 : Set your last-applied from your deployment configuration

kubectl apply set-last-applied -f your-service.yml --create-annotation=true

Step 4 : Check that the last-applied was created, you should see the same values as your downloaded deployment configuration file

kubectl apply view-last-applied deployment/your-service

Step 5 : Edit the last-applied and remove the “replicas:x” line from the file opened and save (vi :wq)

kubectl apply edit-last-applied deployment/your-service

Important : if the last-applied was not created and you try edit-last-applied you will get the missing annotation error.

Step 6 : Enable the HPA, remove the “replicas: x” line from your deployment configuration file and deploy as many times as you want, it shouldn’t ever reset the number of replicas to 1.

This article is inspired from the Github Kubernetes issue : https://github.com/kubernetes/kubernetes/issues/25238

Akli HAMRAOUI
Software Developer Engineer

Thanks to My Team at Egencia - An Expedia Group brand