Modern Django — Part 1: Project Refactor and Meeting the Django Settings API

16 min readFeb 27, 2017

Introduction

For those just joining us, the previous section of the Modern Django guide can be found here: Modern Django — Part 0: Introduction and Initial Setup.

The GitHub for this project and all its sections can be found here.

At this point, we have successfully created a Django project. However, we have a lot of legwork to do before we can even claim it is ready for local development and creating our first application. First, we must extend our project’s requirements. Then, we must extend the project’s settings. We will then create a set of environment variables to hold secret values. Doing these steps will increase the speed at which we can develop, test, and deploy our application in the future.

I suggest you bunker down with some espresso for this section. It’s dense. PC: Alejandro Escamilla

After extending the settings, we will go over the default portions of the Django’s Settings API used by our project when settings.py was first created. We will also meet some commonly used settings not created by default, such as those surrounding the handling of static files. Understanding Django’s Settings API is the first step to shedding light on all that Django Magic™. We will do some more light refactoring and then ensure our project can run!

It is important to make sure all new users are comfortable with Django’s settings, so I apologize to those of you who know Django well or want to get cracking on a REST API. I can only make learning about configuration portions like this so fun (all the fun pictures are at the bottom this time around).

Note: As always transparency is something I value. With that being said, throughout this portion you will notice a similar structure to one provided by the Cookiecutter Django Team. If we didn’t work off of CCD, we would end up just extending the base Django tutorial until advanced portions are needed, so I believe this is a better strategy. The CCD team has figured out a large number of local and production problems and how to mitigate them. I have taken some notes on their structure, compared to where we need to be. While our project may look similar in the early stages, it is important to explain what is being done, instead of blindly creating the template project. Our project will end up being very different in the future, but for now you may notice structural similarities. For example, a large difference we will implement is the removal of server side rendering, (which CCD and many many other projects makes use of) for a REST API coupled with a frontend JavaScript framework.

Temporary Notes: For those of you following along from an older version of Part 0 (pre 2/24/17), we now have an updated project structure. The django-admin startproject command from Part 0 has been updated. Sorry, gang. After looking over my notes a large number of edits were needed to get to the above structure. It is simpler for everyone, following along prior, to remove your project directory and recreate the project under the new name and location.

To do so, move into the modern-django directory, remove the old project, and create the new one. Commands below:

# Commands
cd ~/git/modern-django/
rm -rf project/
django-admin startproject config .

Technology

Not many technologies to read up on for this portion!

Django Settings: “A Django settings file contains all the configuration of your Django installation.” (source). This is the set of files/configurations we will be learning about today! It is the backbone of any Django project.
Environment Variables: “A set of dynamic named values that can affect the way running processes will behave on a computer. They are part of the environment in which a process runs.” (source) We will be making a bunch of environment variables through this process.
django-environ: “Allows you to utilize 12factor inspired environment variables to configure your Django application. “ (source) We will install this package to make managing environment variables with Django easier!

Walkthrough

Picking up from Part 0, our current project structure looks as follows!

modern-django
├── LICENSE
├── README.md
├── config
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
└── requirements.txt

From here on out, I will assume you are in the modern-django directory, unless otherwise specified. To move into the modern-django directory, follow below:

# Command
cd ~/git/modern-django/

1. Extending requirements.txt

In this section, we will refactor the requirements.txt to be far more robust. First, create a directory in the modern-django directory called requirements. Then, copy the current requirements.txt into it and, at the same time, rename it to base.txt. Then, move into the requirements directory and create requirements file for local, production, and test environments.

# Command
mkdir requirements/
mv requirements.txt requirements/base.txt
cd requirements/
touch local.txt production.txt test.txt

Now, we have the ability to change which Python applications our virtualenv (or a Docker image) will use based on the environment (local development, production, or testing).

1-A. Append django-environ to base.txt

In the base.txt, which all environment will build off of, we will add django-environ. This will allow us to simplify our Django Module Settings on two fronts:

Ease usage of environment variables
Simplifying file paths

Open the requirments/base.txt file with your preferred text editor, and make the addition under django==1.10.5:

django-environ==0.4.1

1-B. Ensure Local, Production, and Test Requirements Inherit base.txt

Now, inside of our local.txt, production.txt, and test.txt, we will add a line that will recursively get all of the requirements from base.txt. Add this line to each of the top of the newly created files:

-r base.txt

Just like how we added django to the old requirements.txt in Part 0, this is how we will continue to add more third party applications to the project. More additions will be made to these files as the guide progresses, but for now it is time to turn our attention towards settings.py.

2. Extending settings.py

In this section, we will refactor the settings.py to be far more robust and separate it much like requirements.txt above. We will first create a new directory called settings inside of config:

# Command
mkdir config/settings/

Now, move the settings.py file into the settings directory, and rename it to base.py:

# Command
mv config/settings.py config/settings/base.py

The next step is to create the extended settings for each environment (local, production, and test):

# Command
cd config/settings/
touch local.py production.py test.py

Again, in your text editor of choice, at the top of local.py, production.py, and test.py, extend base.py by adding:

from .base import *

Now that our scaffolding has been made, it is time to walk through and refactor the settings file base.py.

Note: The above line makes use of import *. As a good engineer, you should only import the portions of classes/modules/libraries you need, but with all things, there are exceptions.

3. Meeting the Django Settings API

Currently, the base.py only has the standard settings provided by the django-admin startproject. We will now go down the settings per section; providing a layman’s definition and edits as necessary.

3-A. Adding django-environ

As described in the above sections, the use of django-environ is to simplify environment variable and file path usage.

In base.py, you will find import os under the generated comments. This is currently being used in a few portions to obtain file paths. The use of environ will simplify this functionality for us. Thus, find import os and replace it with importing environ at the top of base.py:

import environ

3-B. File Path Variables: BASE_DIR to ROOT_DIR + APPS_DIR

Now that we have removed import os, we will edit the use of the variable BASE_DIR. BASE_DIR is typically used for building more file path variables off of the project’s top level (commonly known as root directory), i.e. finding another file or a directory of files.

Now that we have environ, we can replace BASE_DIR with another variable called ROOT_DIR. (or simply rename it, if you wish, or don’t, ROOT_DIR just makes more logical sense for the structure). We will then, add the following:

ROOT_DIR = environ.Path(__file__) - 3

This function gets the current files path ~/git/modern-django/config/base.py and moves up three levels to ~/git/modern-django/. Now, we have a variable that points to the root of the project and we can get any other file in the directory from it.

To demonstrate that we can access any other file, we will create another variable called APPS_DIR. This will point it at a folder we have not created yet that will later hold the applications we create. Add APPS_DIR below ROOT_DIR:

APPS_DIR = ROOT_DIR.path('project')

3-C. Import Environment Variables

Below our newly created ROOT_DIR and APPS_DIR, we will import a large number of environment variables to use in the project from a file called .env (we’ll make that soon). We will use the environ.Env() function that will find a .env file in our project root, then read_env() to make use of them. To do so, add the following:

env = environ.Env()# This section added from an update to standards in CookieCutter Django to ensure no errors are encountered at runserver/migrations
READ_DOT_ENV_FILE = env.bool('DJANGO_READ_DOT_ENV_FILE', default=False)if READ_DOT_ENV_FILE:
    env_file = str(ROOT_DIR.path('.env'))
    print('Loading : {}'.format(env_file))
    env.read_env(env_file)
    print('The .env file has been loaded. See base.py for more information')

Notes: The main purpose of adding this functionality in now is so that when using Docker containers, we will be reading environment variables from the .env file, as there are some complications with doing so in Docker.

3-D. Intro to SECRET_KEY and Moving to `local.py`

!!! IF YOU READ NO OTHER SECTIONS, PLEASE READ THIS ONE !!!

Following the addition of environment variables, we come across the setting SECRET_KEY. The secret key in Django is extremely important and like its name suggests, should be kept secret. However, we have already committed our secret key to what most likely is a public repository on GitHub. Don’t worry, this was intentional for local development between environments and other engineers. Note that we will be creating another secret key value when we begin deploying our application, and that value should not be tracked or shared with others.

The secret key is used in a number of places, including creating user’s sessions, messages, cookies, password resets, and cryptographic signing. So, it is pretty important. Read more about it here in the official docs.

For local development, we will cut and paste (removing the value from base.py), the variable SECRET_KEY into local.py and edit the line to match the following (your key value will be different):

SECRET_KEY = env('DJANGO_SECRET_KEY', default='^92l&5_l2f-ik5xlav!7*cat904fro-lmdd@0kgz@c*nxua3@p')

3-E. Intro to DEBUG

We now find ourselves with the DEBUG setting. It is currently set to True. When DEBUG is set to True in Django, a number of things happen, including: detailed error pages, all SQL queries made are stored, and ALLOWED_HOSTS can be empty. Read more about DEBUG here. We should always assume that DEBUG will be False, except in local development where it is True. Edit the current value in base.py to set an environment variable to False.

DEBUG = env.bool('DJANGO_DEBUG', False)

Using DEBUG as True should only be used in local development, and thus we should also make a copy in local.py. Copy and paste the DEBUG setting into local.py and overwrite the environment variable by setting it to True.

DEBUG = env.bool('DJANGO_DEBUG', default=True)

Making a debug environment variable allows us to configure deployment tools more efficiently based on the settings provided!

3-F. Intro to ALLOWED_HOSTS

ALLOWED_HOSTS is the list of sites that Django will accept to make connection and requests. For local development when DEBUG is True, we can have this be empty, however, when in production it must be set to a specific domain such as .example.com.

We will touch on this when we return to deployment. For now, simply cut and paste the ALLOWED_HOSTS into production.py below any imports.

3-G. Intro to INSTALLED_APPS

The INSTALLED_APPS array is the list of applications that the Django project is aware of. It is initialized with a number of default Django applications that run the base project. Typically, projects would simply add in applications as they are installed by pip or created by the engineers. However, for organization will split up the INSTALLED_APPS section into three tuples: DJANGO_APPS, THIRD_PARTY_APPS, and LOCAL_APPS. After we populate the three sections, we will add them together to make the INSTALLED_APPS list.

Note: The usage of tuples, over lists, is because they are immutable and the ease of concatenating them together. Their immutability also ensures, that, at runtime, the project’s installed apps cannot be changed.

So, currently the only applications installed are those default Django apps. Change the structure to reflect this setup:

DJANGO_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
)
THIRD_PARTY_APPS = (
)
LOCAL_APPS = (
)
INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS

As we begin installing more third party applications and create our REST API, we will populate the two empty tuples.

3-H. Intro to MIDDLEWARE

We now find ourselves in front of the illusive MIDDLEWARE. The concept of middleware is taking requests/responses (HTTP communication) as they enter/leave the Django system and applying functions to them before/after being processed in a light weight manner. Such an example is AuthenticationMiddleware, which can associate a user with a request when user sessions are being used. These functions are done extremely fast and help add data we can process against.

IMPORTANT: The order in which middleware are placed in the MIDDLEWARE list is the order in which they will be processed. That means specific middleware must go above others, or else unexpected behavior may occur.

The Django documentation describes this process like layers of an onion, where a View (and its children) is at the center and a request must go through each wrapping layer of middleware in order, before being manipulated by the view.

For a more in-depth explanation on middleware, please read the documentation page.

We can leave this be for now!

3-I. Intro to ROOT_URLCONF

The ROOT_URLCONF points to the URLconf module (a collection of URL guidelines) that Django will handles requests/responses with. Essentially, when a user makes a request to a page/resource in one of our apps, Django looks in that file to determine what to do. That is, the ROOT_URLCONF points to a file that contains a set of URLs such asmodern-django/config/urls.py. The current pointer is the default, and we will add more URLs to the file, such as those to our REST API, later. Currently, you can find it extending to the admin app’s URLs.

3-J. Intro to TEMPLATES

The TEMPLATES setting is a list of settings that are applied to the template engines installed. Multiple can be installed, however most projects will only need one. The default settings include:

BACKEND: the processor for creating templates
DIRS: directories that Django will look for template files
APP_DIRS: set to True, Django will automatically look in each app in the INSTALLED_APPS list for templates that could possibly be rendered
OPTIONS: extra options that depend on the BACKEND
context_processors: an option that is for the DjangoTemplates engine, that determine how data is rendered on a template. Read more here.

More information on settings can be found here.

For the purposes of transparency, we will most likely gut this portion later, as we intend to use a true JavaScript frontend with our API.

3-K. Intro to WSGI_APPLICATION

“WSGI” stands for Web Server Gateway Interface. It is a standard for Python web frameworks that point a server at an application.

WSGI_APPLICATION points to the wsgi.py file that Django will serve when the runserver command is executed. It contains an application as a callable and will point the current settings being used at that application. This is essential for running our application. More information can be found here.

3-L. Intro to DATABASES

The DATABASES settings are those that connect different databases to our project. They store the settings needed to interact with different databases. The default database is a locally stored sqlite3 database (that has not been created yet, as we have done no database manipulation. Yet.). We will later on update this to include a PostgreSQL database when we begin working with Docker.

Again, we have encountered another use of os, thus we should change it to work with a path created by environ. Replace the NAME line as follows:

'NAME': str(ROOT_DIR.path('db.sqlite3')),

3-M. Intro to AUTH_PASSWORD_VALIDATORS

This configuration is a set of rules that passwords must be verified by when created or edited by users. The defaults are robust enough to reduce a large portion of attacks and require no additions. You can create custom validators for increased security as defined here.

3-N. Intro to Internationalization

The section of LANGUAGE_CODE, TIME_ZONE, USE_I18N, USE_L10N, and USE_TZ are all used for formatting data and languages to different users. It has been requested to talk more on this subject at a later date, however for the majority of US / English-based sites, the defaults are more than enough. The documentation for these settings is defined here.

3-O. Intro to Static Files and Media Files

Static files are files that are considered not changing, static. This includes files, such as images, CSS, and JavaScript that are used in the application. Media files are considered dynamic files created/edited/uploaded by users. The majority of rules below apply to both Static and Media files.

We now find ourselves at one of the hardest concepts to grasp for new Django users. There is one standard use case for how static/media file serving should be done, but many users become confused on how to do it.

It is extremely important to understand that static files currently live in separate directories, per app. That location is not the location they will be served from in production.

An example, the CSS files used on the Admin panel will not be served from its own static directory of the admin application in production. The manage.py command, collectstatic, will be used when deploying. This will copy all static files this settings file knows about, into a root level folder defined by STATIC_ROOT. Then, a server, such as nginx, will retrieve and serve those files.

HOWEVER, on local development, Django will simply serve them from the directories they originally live in.

Currently, STATIC_URL is the only static settings created by default. This is the URL that static files will be served at. Currently, the files will be served at localhost:8000/static, if localhost:8000 is used for local development.

We will add the following:

STATIC_ROOT: where the collected static files will be placed
STATICFILES_DIRS: folders Django will look for static files in
STATICFILES_FINDERS: specifications on what files to look for
MEDIA_URL: URL appended to the root URL to serve media data
MEDIA_ROOT: where the collected media files will be stored

Make these additions to Static and Media settings where STATIC_URL is defined in base.py:

STATIC_URL = '/static/'STATIC_ROOT = str(ROOT_DIR('staticfiles'))STATICFILES_DIRS = (
    str(APPS_DIR.path('static')),
)STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)MEDIA_URL = '/media/'MEDIA_ROOT = str(APPS_DIR('media'))

We’re done with configuring settings for now! Hopefully, you have a much better understanding of how all the parts of Django work together and are no longer intimidated by the Settings API! As we install more third party applications, we will add their configurations to the appropriate settings file.

4. Using Our New Settings

To make use of our shiny, new settings, we need to edit manage.py. On the line:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")

Simply append .local to config.settings ensuring it now looks like this:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")

The project will now use the correct settings for local development! Later on, will make additional changes for the production and test settings.

5. Creating a .env File

Way back in 3-C. Import Environment Variables, we discussed how django-environ looks for a .env file, so now we need to make that! In the modern-django directory, we create a file called env.example. This file will be a placeholder for environment variables the project needs. When we use the file for production, we will copy and rename it to .env, and place in the appropriate secret information (such as the real SECRET_KEY). To reiterate, env.example will be tracked by git, .env will not.

Create the example file now, then copy/rename it:

# Command
touch env.example
cp env.example .env

For now, we will leave these files unedited as changes will only be needed when we begin production changes.

6. Update Installed Packages and Runserver!!

Now that our requirements and settings are refactored, it is time to run the project and make sure it all works!

First, install django-environ to your virtualenv from the local requirements. Make sure you activate it before installing, else you will find it on your system’s pip!

# Command
source venv/bin/activate
pip install -Ur requirements/local.txt# Expected Output
...
Installing collected packages: django-environ
Successfully installed django-environ-0.4.1

Now, time to run our server! Use the manage.py command, runserver. It will default run the server at http://127.0.0.1:8000/

# Command
./manage.py runserver# Expected Output
Performing system checks...System check identified no issues (0 silenced).You have 13 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.February 27, 2017 - 04:46:16
Django version 1.10.5, using settings 'config.settings.local'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Navigate to http://127.0.0.1:8000/ or http://localhost:8000/ and you should see:

“It’s alive!” — Dr. Frankenstein (*Young Frankenstein, 1974*)

Note: You will notice the migration issues. We will address those next time.

7. Committing and Pushing

As always, remember to git add your files, commit, and push them to your GitHub. Don’t wanna lose any work.

Summary

We have successfully configured our project to begin local development!

Here is the source code on GitHub for this section.

The structure of the application should be as follows:

modern-django
├── LICENSE
├── README.md
├── config
│   ├── __init__.py
│   ├── __pycache__
│   ├── settings
│   │   ├── __pycache__
│   │   ├── base.py
│   │   ├── local.py
│   │   ├── production.py
│   │   └── test.py
│   ├── urls.py
│   └── wsgi.py
├── db.sqlite3
├── env.example
├── manage.py
└── requirements
    ├── base.txt
    ├── local.txt
    ├── production.txt
    └── test.txt

We have restructured our environment, and are now ready to add in applications. Further amendments will be made as we gear towards development and deployment with Docker.

We will begin next time with a discussion on migrations, furthering user authentication with AllAuth (and a few other tricks), and see if we can fit in the introduction to Django REST Framework.

Really thought I could hold out for a Django Unchained joke.

Until next time, that was a ton of information. Sorry bout that, but we had to get through it. Go grab a beer. We’ll see you back for Part 2 later this week.

As always, feedback, tips, tricks, and criticism are always welcome on the comments or email.

Best and thank you,

Dylan Stein

For updates on Modern Project please check out:
MODERN PROJECT / NEWSLETTER / PATREON