Turn Off Your Applications Safely: Graceful Shut Down and Signals

Shut Down option in MacOS

When we first get to touch computers, we were taught to turn off the computer by pressing the “Shut Down” button instead of directly unplugging the computer from its power source. But why exactly?, we get the same results anyways which is the computer getting turned off. One good example why can be seen in the screenshot below.

Notification if you try to close Microsoft Word without saving your changes

Problem Statement on why Graceful Shutdown is needed

The simple reason is that the computer might still have tasks in progress, if you turn it off suddenly then task would be interrupted, potentially causing problems due to unfinished operations.

The same can be said with your applications, but instead of the computer / server, you turn off your application. Suppose you have an e-commerce app, obviously you wouldn’t want to suddenly interrupt a payment in process for your customer’s purchase and leave them clueless on why wouldn’t their payment finish. You want to wait until their payment is finished, and only after that you would want to turn off your app.

Situations where you need to turn off your app

With the advent of the internet and how it’s highly accessible nowadays, applications are mostly web-based which means the app is running in your server and requires it to be turned off whenever you want to update it. On top of that with CI/CD culture, you should expect multiple deployments for your app in just 1 day, which also means you would also need to turn off your app as many times as you deploy it.

Terminology

Before getting into the discussion, let’s agree w/ these definitions:

• Graceful Shutdown: shutting down an application “safely” by ensuring that the internal on-going processes have been finished and all connections are closed
• Signals: OS signals sent to a given process to indicate events (such as termination, stop, kill, etc. you can see the full supported signal list by running kill -l in your UNIX server / machine)

How a graceful shutdown is done in high-level for an application

1. Close all of the app’s entry point to receive new tasks (remove the server from an LB’s target group, unsubscribing from a message queue, etc.)
2. Let your application know that it needs to be turned off soon (step 1 and 2 is interchangeable since some mechanism needs the app to know that it needs to be shutdown first)
3. Application gets to know that it needs to be turned off soon, and finishes its’ remaining tasks
4. Application finished its’ remaining tasks, and shuts down
5. Application is shut down gracefully with no interrupted tasks

There are multiple ways of doing these 5 steps obviously, but for simplicity in this article we’re going to only discuss the most common and “standard” way, specifically using SIGTERM OS signal to indicate that the process (application) is going to be shut down.

How do applications receive signals?

Every application that you run is a process. Whether it’s as simple as a 10-liner Python script or a giant Java app with thousands of classes and multiple connections to database servers. Going back to the signals definition that we have discussed before, this means we can send signals to our applications’ process to indicate that it needs to be turned off soon.

OS Signals are not coupled to any programming languages, you should be able to find a pre-made library for your programming language by searching “<programming language name> signal handling” or something like that. But for better understanding, here’s an example of handling SIGTERM signal written in Python

Sample code in Python to handle SIGTERM signal

Note: you might see 2 unused variables in sigterm_handler() function, that is due to the library’s implementation where the handler function is called with 2 variables as stated in the docs. Adjust this with the programming language and the library that you are using!

Now if I run the “application” that I wrote above, obviously I would be stuck in an infinite loop. But I have defined a function called sigterm_handler() to be run when it receives a SIGTERM signal, to manually send a SIGTERM signal, we can use the command kill -s TERM <process id> on UNIX systems. Below is the demonstration of the said command

Demonstration of using SIGTERM to the application mentioned above

Note: ps aux | grep simple_sigterm_handling means to find all processes that has "simple_sigterm_handling" in its’ details, we used this to find the application’s process ID, in this case there are 2 matching processes, the application itself, and the grep process we used. Read more about ps aux and its output here

As you can see, the Python application that we have run has shut down gracefully as before shutting down since it was able to know to handle remaining tasks before shutting down which in this case was to run the sigterm_handler() function.

How SIGTERM is used in the real world scenarios

As we have seen in the previous section, it’s already clear on how we do it, but it seems to be very manual, and do we really have to run kill -s TERM <PID> every time we want to shut down our applications gracefully? Luckily the answer is no as most of the time your application is run using service managers such as systemd (and also Docker and Kubernetes if you count them in too as service managers) that also utilizes SIGTERM as a graceful shutdown mechanism.

Assuming you already understand on what is systemd and using the commands related to it (service or systemctl), the official documentations also state that the default behaviour when stopping a service is to send a SIGTERM signal first, and then wait for the app to shut down or if 90 seconds have passed then it would force shut down (using SIGKILL signal). You can see below screenshot using the exact same application for demonstration.

You can see that systemd sent a SIGTERM first when we stop the application

The exact same pattern is also used in Docker (docs) and Kubernetes (recommended article), but in Kubernetes you can also specify additional commands to be run during the period the SIGTERM is being handled.

Conclusion

Always gracefully shut down your application, whether it’s for your server’s maintenance or deploying a newer version of your application since you don’t want any of the application’s unfinished tasks to be suddenly interrupted. The most common way to do this is by giving SIGTERM signal handling to your applications.

References

[1] https://www.techtarget.com/whatis/definition/graceful-shutdown-and-hard-shutdown

[2] https://www.tutorialspoint.com/unix/unix-signals-traps.htm