Scheduling Tasks in Django with the Advanced Python Scheduler
Scheduling tasks for the future is an essential tool for any software developer. While much of the programming we create aims to respond to explicit triggers or user events, background processes performed at regular intervals are just as essential.
“Update the results each Monday morning.”
“Batch our orders every night.”
Even third-party APIs with daily request limits implicitly demand this type of behavior.
“We can only request an update every five minutes.”
Fortunately, this problem has been tackled by a lot of smart people, and a python-native solution is not tough to find. The Advanced Python Scheduler (APS) is a great option which features an easy, intuitive API, as well as some of the best documentation in its class.
For this project, we will focus on integrating the scheduling technology that APS offers with your run-of-the-mill Django app: a Los Angeles Weather App which regularly polls a third-party weather api for model updates.
The goal is to take a deeper dive than you typically get with a Django tutorial, while not getting too caught up in the weeds in any direction.
I. Install APS and other dependencies
In your project directory, create a virtual environment and activate it
virtualenv env
. env/bin/activateInstall and configure PostgreSQL according to this guide. At this stage, we only need the SQL manager up and running on your machine.
Additionally, I find it helpful to use the PgAdmin PostgreSQL GUI. Details on setting this up on your machine can be found here (use Python3).
Use pip to install all required packages (note, psycopg2 is for PostgreSQL):
pip install apscheduler django psycopg2 requestsII. Build your app
Create a new Django project:
django-admin.py startproject advancedScheduler
cd advancedScheduler
python manage.py startapp weatherIn this new directory (the Root Directory), you will see another folder named advancedScheduler. This is the Django Project Directory.
To avoid any same-name-in-two-places confusion, we’ll only refer to the Root Directory as Root Directory. Let the diagram below serve as a road map on our folder-jumping adventure.
[ super_project_directory/ ]
|
+----[ env/ ] <-- Virtualenv stuff
|
+----[ advancedScheduler/ ] <-- the Root Directory
|
+----[ advancedScheduler/ ] <-- the Django Project Directory
|
+----[ weather/ ] <-- the Django App DirectoryThough principally focused on demonstrating the power of our scheduler, let’s take a moment to wire up our Django App.
We’ll first want to add the weather app to our project’s INSTALLED_APPS. That’s in the advancedScheduler/settings.py file.
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'weather'
]Next, add a new url pattern to the advancedScheduler/urls.py file:
path('', include('weather.urls'))Not surprisingly, our next step will be to add that urls.py file to the weather app directory. Include the following code to weather/urls.py:
from django.conf.urls import url
from weather import viewsurlpatterns = [
url(r'^$', views.MainPage.as_view())
]
Create a templates folder in the weather app directory. Add an index.html file to this new folder.
Below is our MTV.
Model
Template
View
III. Establish the DB connection and migrate your models
In the advancedScheduler/settings.py change the DATABASES value to:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'advancedScheduler',
'USER': 'some_user_name',
'PASSWORD': 'some_password',
'HOST': 'localhost',
'PORT': '',
}
}You should know your USER, PASSWORD, and PORT values from the above-mentioned PostgreSQL configuration guides (here & here).
With the connection with PostgreSQL established, it is time to migrate our models. Navigate to the Root Directory and type:
python manage.py makemigrations
python manage.py migrateWith that, our models should have been mapped to the database. Go ahead and verify everything’s where it should be. Don’t worry, I’ll be here waiting for you to get back.
IV. Forecast API
Time for the fun part. I’m pulling my forecast data from OpenWeatherMap, a free weather API that will grant you an access token with a valid email address.
Now, since it’s conceptually distinct from our presentation layer, let’s create a new forecastUpdater folder in the root directory. In it we will add two files: A blank __init__.py file, and a forecastApi.py file. See the roadmap for reference.
[ super_project_directory/ ]
|
+----[ env/ ]
|
+----[ advancedScheduler/ ] <-- the Root Directory
|
+----[ advancedScheduler/ ]
|
+----[ weather/ ]
|
+----[ forecastUpdater/ ] <-- the new Updater Module
|
+----< __init__.py > <--+
| |-- two new Python files
+----< forecastApi.py > <--+Here, there are some things to note. The exception handling is far from robust. Errors are just dropped — excessive silence is the only indication something is wrong.
Second, we specify Los Angeles in our code. Configure yours to whatever location you’d like.
It is also important to note that update_forecast() takes no parameters. We’ll soon see that our advanced python scheduler is rather strict with its no parameter rule. Not even methods with a lonely self parameter will fly.
V. Advanced Python Scheduler
We’ve built our models. We can update our data with calls to our API. All that we need to do now is specify how frequently to access that API, so we can provide reasonably up-to-date information while falling within our data-access limit.
In the forecastUpdater module, add an updater.py file. Here is where we will use the Advanced Python Scheduler to set the cadence of our forecast updates.
OpenWeatherMaps terms of use allow for 60 calls in an hour to remain in the free tier; an update every five minutes should be more than adequate.
This is possibly the most simple APS implementation you could find. If you check out their site, or the several working examples on GitHub, you will find a whole toolbox of features and settings you can use to get as meticulous as you want with your timing.
Once our scheduler is configured just the way we want, it’s time to wire it up to our Django app.
Ideally, we are looking to press play on our scheduler once and let it do its thing. We want a consistent and reliable way to initialize a schedule once and only once. Conveniently for us, Django has just the place for this type of runtime-initialization logic.
In the weather/apps.py file you will find a stub of a class called WeatherConfig which inherits from Django’s AppConfig class.
class WeatherConfig(AppConfig):
name = 'weather'To let Django know it needs to kick off our updater on startup, we overwrite the AppConfig.ready() method.
It is important to remember that, due to the intricacies of inheritance, any imports for this override must be located within be body of the ready() method. Django also warns against interacting directly with the database in our override; production, debug, rain or shine, every time the weather app starts, this code will execute.
Lastly, we need to now update INSTALLED_APPS variable again in our advancedScheduler/settings.py. Django needs to know that we want to run our weather app with a custom configuration.
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'weather.apps.WeatherConfig'
]VI. Putting it all together
That’s it. At this point we can start up our app and let the updater do its thing.
python manage.py runserver --noreloadThe --noreload flag keeps Django from starting up a second instance of our weather app — which is the default behavior in debug mode. A second instance would mean all of our scheduled tasks would fire twice.
Initially our result will look incomplete. Because we scheduled our updater logic to run every five minutes, we’re stuck twiddling our thumbs for a bit… To spice things up, it might be prudent to shorten the interval between scheduled refreshes, or call update_forecast() once on initialization.
VII. Final Thoughts
We did it! Our weather app is ready to share it with the world (see mine here).
The Advanced Python Scheduler is a great tool for any python developer to know about. It hides the complexities of an all-too-common business requirement behind an intuitive API. Think about it, setup only took three lines of code.
The real trick in this project is interacting with the Django framework — configurations, migrations, initialization. Task automation then becomes an afterthought. You’re finished with it in five minutes.
Thank you for taking the time to go through this tutorial. I hope it was some help. If you have any questions, comments, or suggestions, please don’t hesitate to let me know.
See the source code on GitHub.