Graceful shutdown in go with Kubernetes

In this post, I will explain the Kubernetes termination lifecycle and how to implement a graceful shutdown in Golang.

Graceful Shutdown

There are two types of shutdowns, one is a hard shutdown and the other is a graceful shutdown. Consider this example, when you unplug your desktop computer, it shutdowns abruptly this is a hard shutdown, but when you do it through your operating system this is a graceful shutdown. OS can save the state to the disk and clean up the processes since it knows it is going to be shut down.

So, by implementing graceful shutdown in our code, we are telling the service that you are going to be shut down soon, so try to finish these requests without abruptly closing them, or saving the state to the disk to prevent data loss.

Kubernetes Termination Lifecycle

Once Kubernetes decides to terminate the pod, the following events will take place.

Firstly, Kubernetes sets the pod to the “Terminating State”.

Secondly, if you added a preStop hook to your deployment file, Kubernetes will run it before sending the termination signal.

This sleep can be used to delay the SIGTERM signal.

Thirdly, after the preStop hook is executed, a SIGTERM signal is sent to the pod. The application should be listening to this termination signal so that it can gracefully shut itself down.

And then, Kubernetes by default waits for 30 seconds for the termination grace period which can be changed to something else in your deployment file.

Note that this termination grace period starts in parallel with the preStop hook. So if your preStop hook fails, it does not care. After a given value, Kubernetes proceeds.

Finally, a SIGKILL signal is sent to the pod and the pod is removed.

Termination Lifecycle

Handling graceful shutdown in the Golang application

As it can be understood from the above lifecycle, we need to be able to catch the SIGTERM signal and start a graceful shutdown within our application.

For this purpose, there is no need to catch another termination signal. Kubernetes is sending SIGTERM and finally SIGKILL to shut down the pod. SIGKILL itself is not catchable. So, all we need to do is catch SIGTERM and close our application gracefully. And for local debugging purposes, you can also add SIGINT.

For example,

Graceful shutdown of the net/http server

If you put this code block under the <-gracefulStop, then your HTTP server will shut itself down after SIGTERM is sent to the pod.

The Two Forms

Even if you do everything as explained above, your application might not be able to receive SIGTERM signals. The reason behind this is another topic.

Take a look at these Docker files.

Shell form on the left, Exec form on the right

They look essentially the same except for the quotation marks and brackets. This is the most important thing we need to check when implementing the graceful shutdown.

What is the difference between these two? Why can one of them do a graceful shutdown and why the other cannot receive the signal at all?

Shell form commands are run by the container’s shell. Most shells are running these processes as child processes, and they do not forward signals to the child processes. This makes our application unable to gracefully shut down because it has no idea since the signal is not sent at all.

We don’t have this problem for the exec form, but we are sacrificing some benefits of the Shell form such as some Shell features, piping, etc.

So, when doing this kind of implementation, always keep in mind that your application needs to be the parent process to receive the signal sent from Kubernetes.

Thank you for reading this article, if you have any questions feel free to ask them below.