An elegant way to run periodic tasks in python

At GreedyGame, we try to help Game Developers monetize without hampering the gameplay experience of the Gamer. We work towards creating a better ad ecosystem by bringing native ads to mobile games and creating Ads people ❤

Periodic tasks are tasks which are executed at specified time intervals, again and again, with minimal human intervention. Every engineer would have used it in one way or the other, to schedule periodic database backups, enriching data for your Machine Learning algorithm, polling an API, daily marketing emails, etc.

I was working on a module which needed me to keep some entries in a database table for a certain period of time post which the entries should get deleted automatically. In order to do this I wanted to go with the following approach:

1. Include an expire_at column in the table
2. Run a periodic task that deletes all entries from the table where expire_at < current time

The most common approach is to use a cron job, however using a cron will have a dependency on an external process that is not part of your code base which will have to be managed separately. I wanted something that could be accessed within my package.

With python, there are several ways of creating and executing a periodic task. Some common ways are:

1. Celery beat
2. Using time.sleep
3. Using threading.Timer
4. Using threading.Event

1. Celery beat

Using celery beat, tasks can be started at regular intervals which would be picked up by available workers in a cluster.
This is a heavy framework with such level of complexity which is not required for most types of use cases. For my specific use case I needed a lightweight utility.

2. Using time.sleep

One could run a simple loop with whatever duration you want in time.sleep

With this approach, if the program is killed in between, the function foo() would be killed abruptly. Also if you want to run multiple tasks at different periods you would have to do that separately in another script.

3. Using a Timer

A periodic task could also be implemented using the Timer on a thread.

This approach has the same problems that exists in the time.sleep approach. Instead we can use the threading.Event module.

4. Using threading.Event

threading.Event module has a flag that can be set or reset at any point of time using set() and clear(). It also has a wait() method which blocks until the flag is set. This functionality can be used to created a periodic task.

Need for a lightweight approach

In all the previous methods, there is no way to guarantee a graceful shutdown of tasks, in case the program is killed or has ended abruptly.

When a program has to be shut down gracefully, it is advised to do it via a soft kill which sends a SIGKILL signal to the program. The program can then intercept this signal and run some cleanup code before exiting. It could also intercept a SIGINT signal which is sent when you stop running a program via the ctrl+c command.

In python, this can be done using the signal module which can intercept the above signals. But the tasks must run in separate threads so that the main thread can be used to catch the signals and run the cleanup code. Please refer to this code snippet for context.

Hence I wanted to design a library that had the following capabilities.

1. Run multiple periodic tasks at different intervals.
2. When a program is killed, all tasks that are currently running must be completed before the program exits.
3. The library should be simple to implement and use. ( I went with a decorator pattern, which is a good way to attach additional responsibilities to an object dynamically. This is a better alternative than to extend functionality using a subclass.)

Introducing timeloop

timeloop is a library which can be used to run multiple period tasks. It uses a decorator pattern and can be integrated easily in any existing code base, or used as a stand alone program.

All jobs declared by the decorator will run as separate threads.

Installation

pip install timeloop

If this program is killed or interrupted, it will wait for all the jobs that are currently running before exiting.

sankalpjonn/timeloop

Happy looping!