- How to propagate a change in Kubernetes Secrets by restarting dependent pods!
So, you find yourself in a situation where some API tokens or Access Keys of some of your applications have changed and you have to update your Kubernetes Secrets to accommodate the change and load new values — but all the running pods that depend on the secret hold on to the old values unless they are restarted!
I came across this GitHub issue which tries to find a solution for the same but it is closed in favor of an Open Feature Request that you can track here. There is also a SO question that hints at a potential solution, which we will be exploring.
Yes, you can perhaps install some custom K8s controllers and use annotations that add dependencies and tracks the changes and does the update, but this blog is about a simpler alternative that runs imperatively and might work for quite a bunch of use-cases.
Enough intro, let’s see some code!
But first things first. Let’s set the stage. Assume you have a secret named postgres-credentials and has the keys DB_HOST,DB_PORT,DB_USER,DB_PASS and DB_NAME which your application uses to create a connection with the server. Now, for some reason, you are changing the DB_HOST, say to a PaaS service and now all the applications should start using the new database server. You can write the below code to delete and create the secret from literal (or use a Secret.yaml)
Now that you have updated the secret, the idea is to find all the pods that are depending on it and do a kubectl delete pod <pod_name> But what if you have several applications and not sure how many pods are depending on the secret! Thanks to Kubernetes jsonpathoutput formatting and with a bit of bash magic, this is easy! The code was inspired from the SO answer here
And there you go, you get a list of all pods that use the secret as a reference, you can do a simple ./FindDependentPods.sh postgres-credentials | xargs -n1 kubectl delete pod to delete the pods and the corresponding Kubernetes Deployments or Replication Controllers will spin them back up with the updated values!
It might not be the most ‘elegant’ solution since you are basically killing the application till the time pod gets back up and running, but incorrect values for secrets wouldn’t keep the application functional either! You could consider adding a sleep to kill pods over a period.
If you have suggestions, please comment! You can find me on LinkedIn
