Give your kubernetes pods a lifetime
Have you ever encountered a use case for your kubernetes setup where you wanted your pods to have a maximum lifetime? Possibly have a controller that automatically reaps (kills!) pods that have reached their maximum age?
We had this use case and this was very specific to our mutual TLS setup.
We use HashiCorp Vault to issue certificates to the individual pods which acts as the client identity in a mutual TLS setup. The pods send this certificate when making service calls to a mTLS enabled service.
The certificate issuance is configured via a init container in the pod spec. This init container connects to Vault with the necessary token and certificate signing request (CSR) to generate the certificate. As part of our security policy, Vault sets the expiry of this certificate to 30 days from the time of issuance. Consequently, any requests made from the client pod to the service after the 30 day period would fail. This is where the requirement for having a lifetime for our pods comes from. Since the certificate issuance only happens during pod initialization, we need to make sure that a certain pod lives only for 30 days max. Usually, the developer teams roll out a release every other day so this is not the common case that pods live for that long, but this does happen once in a while that a rollout has not been for 30 days.
So how do we solve it?
We searched the kubernetes spec to see if there was any way to do this. But there wasn’t. So we ended up writing a simple controller that is in the same cluster as the pods and reaps all pods that have reached their lifetime. Find the git source for this controller here :
To give a lifetime to the pods, we add the following annotation:
pod.kubernetes.io/lifetime: $DURATION
DURATION has to be a valid golang duration string.
A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as “300ms”, “-1.5h” or “2h45m”. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”.
Example:
pod.kubernetes.io/lifetime: 720h
The above annotation will cause the pod to be reaped (killed) once it reaches the age of 30d (720h). For our case, we usually set the age to 29d just to give us enough room before the 30d cert expiry.
The controller can be configured via variables to reap only certain namespaces as also the maximum pods to reap in one run. Check the README for more details.
Important: The controller does NOT do a rolling update at the moment, so it can potentially cause downtime for your apps. It’s trivial to change the code to do a rolling update for the replicaSet instead of deleting the pod, but I have not got to it yet.
Leave a comment if you have encountered the same issue in your setup!
