Django DevOps Part 1: Dockerizing Django project

Welcome to this four-part series on Django DevOps! This beginner-friendly guide covers everything from setting up a dockerized Django project for your local development environment to deploying it on Amazon EC2, and finally, integrating CI/CD pipelines to streamline your development workflow. I will explain each step in detail to ensure you understand the entire process thoroughly.

Part 1: Dockerizing our Django Services

In the first part of our series, we will start by dockerizing various services of our Django project. I will show how we can break our project down into services and also discuss the basics of docker file, docker-compose, volume, and docker network.

Part 2: Creating a virtual machine and configuring the DNS for our domain.

We’ll cover everything from configuring the DNS record of a domain, and setting up an EC2 instance in AWS. (View Part 2)

Part 3: Deploying on Amazon EC2 with SSL

With our Dockerized services ready, we’ll move on to deploying our Django application on the Amazon EC2 instance. Additionally, we’ll use a dockerized Certbot service to automatically obtain and renew SSL certificates. (View Part 3)

Part 4: Integrating CI/CD Pipelines

In the final part of our series, we’ll focus on adding Continuous Integration and Continuous Deployment (CI/CD) pipelines to your project. Our CI/CD pipeline will automate the process of building, testing and deploying our code.

Whether you’re new to Django, Docker, or DevOps, this tutorial series is designed to help you understand the building blocks with ease. So, let’s get started!

Our Docker Services

Each of our services will operate in different containers. Think of them as mini virtual machines. They will play different roles and will communicate with each other with the docker network once deployed.

1. django_Gunicorn service: We will have our Django project repository in this service. Django is built with Python and a typical web server (Nginx in our case) can’t talk to Python code natively. As such, we will have Gunicorn as an intermediary between the Django application and the Nginx web server.
2. Nginx service: Nginx is a high-performance HTTP server that acts as the first point of contact for client requests. It splits these requests, forwarding regular operations (like CRUD operations) to Gunicorn, while serving static files (CSS, js, images) requests directly by Nginx itself.
3. PostgreSQL service: This service will host our PostgreSQL database. When Django receives a CRUD request it communicates with this service to process that data.

We will write Docker files for each of them and we will also have a central Docker compose file for managing these Docker services.

1. django_gunicorn Docker Service

Why do we need Gunicorn?

A lot of things happen when a request comes. Serving static files, returning dynamic responses, load balancing for a large number of requests, and so on. Having Gunicorn helps us to divide these tasks.

We streamline tasks by having Nginx handle external requests and serving static files (like images, CSS, and JS files) directly. This takes some load off from our Django project.

Gunicorn’s role is then simplified to processing other requests and passing them to Django. Gunicorn also makes load balancing possible. Gunicorn can run multiple worker processes. During heavy traffic, Nginx forwards these client requests to Gunicorn, which then processes the requests in parallel across its workers.

Let's look at the following Diagram for a better understanding:

Figure 2: How the docker services communicate with each other

You can see how the Nginx service (2) only serves static and media files by itself but forwards other types of requests to our Django Gunicorn service (1) with reverse proxy. We will explain the other parts of this diagram later.

At the end of this tutorial, I have also included my project repository.

So, let's build our django_gunicorn docker service!

Dockerfile

A Dockerfile is used to build the container image that will run our services. For the django_gunicorn container, the Dockerfile will be placed in the root directory where manage.py is located.

FROM python:3.10-alpine
COPY . /app
WORKDIR /app
RUN pip install --upgrade pip
RUN apk add postgresql-dev
RUN pip install -r requirements.txt
RUN chmod +x entrypoint.sh
ENTRYPOINT ["sh", "entrypoint.sh"]

1. In the first line, we specify the base image for our container.
2. Then we COPY all files from the current directory on our local machine (where the Dockerfile is located) into the /app directory inside the Docker container. This includes the Django project, Django apps, and other configuration files. I have also written a ‘.dockerigonre’ file to exclude some of the files and directories (e.g., the nginx directory is not needed).
3. WORKDIR sets the working directory inside the Docker container to /app. Any subsequent commands will be run in this directory and we don’t need to ‘cd’ into the directory.
4. Then we update our pip package manager and install a package called ‘postgresql-dev’. This package is required to interface with a PostgreSQL database, like psycopg2.
5. We install all of our required Python packages from the ‘requirements.txt’ file.
6. Using the last two commands we set execution permission for a bash script and we set it as the ENTRYPOINT. When docker-compose runs this container this script will run first.

entrypoint.sh

An interesting thing to notice here! When building a container image, the Dockerfile runs only once, and the cached image is reused for efficiency each time the services are started. However, the ENTRYPOINT script in the Dockerfile runs every time the container is started, so it should contain commands that need to execute after each update and deployment.

#!/bin/sh

python manage.py migrate --noinput
python manage.py create_super_user
python manage.py collectstatic --noinput
gunicorn project.wsgi:application --bind 0.0.0.0:8000

1. In the first line, we do database migrations. The ‘ — noinput’ flag ensures that this command runs without needing any user input.
2. I have written a custom Django management command for creating a superuser if there is none. Here I am calling that command. You don’t need this if you are not using Django admin.
3. Next, we collect all the static files (e.g., CSS, JavaScript, and images) from each app in our Django project into a single location specified by the STATIC_ROOT setting in our Django project. Later we will define docker volume to link this directory to another directory inside the Nginx container. It's like having a pipeline between two containers for accessing shared files. This way, the Nginx service will be able to serve them.
4. Lastly, we start the Gunicorn web server, which is a Python WSGI HTTP server for running our Django application.

• project.wsgi:application specifies the WSGI application callable that Gunicorn should use to serve our Django project. In my case, the name of the project is also ‘project’, if you are using a different project name, replace ‘project’ with that.
• --bind 0.0.0.0:8000 tells Gunicorn to bind to all available IP addresses on the server (0.0.0.0) and listen for incoming connections on the port 8000. Later we will use this 8000 port in the reverse proxy setting of our Nginx configuration so that Nginx can forward the requests to Gunicorn.
• Please note that in my requirements.txt file, I have added the Gunicorn package. You should do the same.

requirements.txt

asgiref==3.8.1
Django==5.0.3
django-ckeditor==6.7.1
django-js-asset==2.2.0
psycopg2==2.9.9
python-dotenv==1.0.1
sqlparse==0.4.4
tzdata==2024.1

gunicorn==22.0.0

create_super_user.py

from django.contrib.auth.models import User
from django.core.management.base import BaseCommand, CommandError
from django.conf import settings
from django.db.models import Q

class Command(BaseCommand):
    def handle(self, *args, **options):
        existing_admin = User.objects.filter(Q(username=settings.SUPERUSER_USERNAME) | Q(email=settings.SUPERUSER_EMAIL)).first()
        if(existing_admin is None):
            superuser = User.objects.create_superuser(
                username=settings.SUPERUSER_USERNAME,
                email=settings.SUPERUSER_EMAIL,
                password=settings.SUPERUSER_PASSWORD)
            superuser.save()
        else:
            print('Super user already exists !')

This is my custom command for creating a superuser that I run from the entrypoint shell script.

2. Nginx Docker Service

For our django_gunicorn service, we had our Dockerfile in the root directory. However, for the Nginx docker service, we will keep our files in the Ngix directory inside the root directory.

Nginx directory for keeping required Docker files

Like django_gunicorn we have a Dockerfile for building our container. We also have a ‘dev.conf’ file inside the ‘conf’ directory where we will define our configuration.

dev.conf

upstream django {
   server django_gunicorn:8000;
}

server {
   listen 80;

   location / {
       proxy_pass http://django;
   }

   location /static/ {
       alias /static/;
   }

   location /media/ {
       alias /media/;
   }
}

• The upstream block in Nginx specifies where to send requests for the Django application. The name django_gunicorn isn't an IP address; it's the Docker service name for the Django-Gunicorn container, as defined in the docker-compose file (we will create that at the end). Docker resolves this name to the container's IP address. This technique is also used to connect Django-Gunicorn with the PostgreSQL service.
• The server block configures Nginx to listen on port 80 for incoming HTTP requests.
• The location/ block proxies all requests to the Django application running on Gunicorn. Note that we used the same name ‘django’ in the upstream block and location/ block.
• The location /static/ and location /media/ blocks serve static and media files directly from the file system, improving performance. We will use docker volumes to link these directories to the static and media directories of the Django project.
• Please note that this is the Nginx service. So, don’t use the value of your MEDIA_ROOT and STATIC_ROOT for these Nginx configs. Django and Nginx are in two different containers and are isolated. We will link them later.

Dockerfile

FROM nginx:1.23.0-alpine
COPY ./conf/default.conf /etc/nginx/conf.d/default.conf

1. First, we define our base image. It is an Nginx web server version 1.23.0, using a lightweight and secure Alpine Linux distribution.
2. The custom Nginx configuration file from the host machine directory 'default.conf’ is copied into our container ‘/etc/nginx/conf.d/’ directory. This is the directory from where Nginx read config files.

3. PostgreSQL Docker Service

Unlike the previous two services, we will not have a Dockerfile for the PostgreSQL service. Instead, we will define the base image directly to our docker-compose.

So why we don’t do that for the previous two services as well?

It’s because we are not modifying the base image for postgreSQL service. We are also not running any custom commands or copying any files into the container. Also, the docker-compose file allows us to configure the base image to some extent and we will do that.

So let’s write our docker-compose file!

docker-compose.yaml

version: '3.8'
services:
  postgresql:
    image: postgres
    volumes:
      - postgresql-db:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    env_file:
      - ./.env
    environment:
      - POSTGRES_DB=${DEV_DB_NAME}
      - POSTGRES_USER=${DEV_DB_USER}
      - POSTGRES_PASSWORD=${DEV_DB_PASSWORD}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DEV_DB_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5

  django_gunicorn:
    build: ./
    volumes:
      - static:/app/project/static
      - media:/app/project/media
    env_file:
      - ./.env
    ports:
      - "8000:8000"
    depends_on:
      postgresql:
        condition: service_healthy

  nginx:
    build: 
      context: ./nginx
    volumes:
      - static:/static
      - media:/media
    ports:
      - "80:80"
    depends_on:
      - django_gunicorn

volumes:
  static:
  postgresql-db:
  media:

We will place this file into the root directory where we have our manage.py file. Three service blocks in this file refer to all three of our docker services (postgresql, django_gunicorn, and nginx). There is also a volume block at the bottom where we have defined three docker volumes. How about we break it down:

postgresql:

This block contains our postgreSQL container configuration. The top line of this block is the container name.

• The PostgreSQL service uses the official postgres image.
• A Docker volume is used to persist the database data.
• Port 5432 is exposed for database access.
• Environment variables are used to configure the database name, user, and password, with values sourced from an external .env file for flexibility.
• The ‘healthcheck’ block in the postgresql service uses the pg_isready command to check if PostgreSQL is ready to accept connections. It runs every 10 seconds and retries it up to 5 times. We will use it in the next service block to ensure the django_gunicorn service waits until postgresdb is ready.

django_gunicorn:

‘django_gunicorn’ is the name of the service. We used this name in the config file upstream block.

• The build directive tells Docker Compose to build a Docker image for this service using the Dockerfile located in the current directory ‘./’.
• Static and media files are stored in Docker volumes, the same volumes are also used in Nginx. If you are using your own project put values of your MEDIA_ROOT and STATIC_ROOT over here. In the case of my path, the ‘app’ directory is our container root directory that we defined in the django_gunicorn’s Dockerfile.
• The service uses an external ‘.env’ file for environment-specific configuration.
• Port 8000 is exposed to make the container accessible from the host machine.
• The service depends on the PostgreSQL database service, ensuring it starts and is ready to connect before the Django application.

nginx:

• The Nginx image is built using a Dockerfile located in the ./nginx directory, which likely contains custom Nginx configurations.
• Volumes: The Nginx service accesses static and media files via shared volumes, allowing it to serve these files directly.
• Ports: Port 80 is exposed to allow HTTP traffic to the Nginx server, making the application accessible to users.
• Depends On: The Nginx service is dependent on the Django application service (django_gunicorn), ensuring that Nginx only starts after the Django application is running.

How does shared Volumes work?

We defined some common volumes in the volume block and then used those volumes in different services. The syntax looks like this:

services:
  service_x:
    volumes:
       volume_a: /path/to/a/directory
  service_y:
    volumes:
       volume_a: /path/to/a/different/directory

volumes:
   volume_a:

This setup is called named volumes. On the left side, we define the name and on the right side, we provide the directory for that particular container. When we use the same name for two containers they get linked together allowing inter-container data sharing.

Where is the .env file, how does it work?

As you have already seen, I have referenced a ‘.env’ file in my docker-compose and also inside my Django project repository (dev.py, prod.py, settings.py). For simplicity I wanted this whole setup to have one .env file which I can use in various places.

The docker-compose reads this file in the host like a regular file.

The django_gunicorn container’s Docker file has a COPY statement which copies the all required files including the ‘.env’ file so that our Django repository (dev.py, prod.py, settings.py) can also read it.

I didn’t include the .env file in the GIT repository but I have provided a ‘.env.example’ file which you can use as a template for creating your own ‘.env’ file. In this tutorial, we are only using the DEV variables from that file.

.env.example

# Create a files in this directory named as '.env' following this example. 
# Copy example code from this file into your own file and fill in the values.
# Change the value of the 'IS_DEV' variable depending on whether you are in production mode or development mode.

SECRET_KEY='replace_this_key_with_a_secure_key'
IS_DEV='True'

# Development only
DEV_DB_NAME='my_db'
DEV_DB_USER='postgres'
DEV_DB_PASSWORD='my_db_password'
DEV_DB_HOST='postgresql'
DEV_DB_PORT='5432'
DEV_SUPERUSER_EMAIL='my_email@example.com'
DEV_SUPERUSER_USERNAME='my_username'
DEV_SUPERUSER_PASSWORD='my_superuser_password'

# Production only
PROD_DB_NAME='my_db'
PROD_DB_USER='postgres'
PROD_DB_PASSWORD='my_db_password'
PROD_DB_HOST='postgresql'
PROD_DB_PORT='5432'
PROD_SUPERUSER_EMAIL='my_email@example.com'
PROD_SUPERUSER_USERNAME='my_username'
PROD_SUPERUSER_PASSWORD='my_superuser_password'

PROD_ALLOWED_HOST='my_domain.com'
PROD_CERTBOT_EMAIL='my_email@example.com'

I have included the Github repository for this tutorial. Download the project, create your .env file and use the following steps to start our dockerized Django project:

1. Install docker, docker-compose
2. Clone the project: git clone https://github.com/romy47/portfolio.git
3. CD into the project directory: cd portfolio
4. Switch to the appropriate branch: git checkout medium_1_docker_dev
5. Create a ‘.env’ file following the example given on ‘.env.example’
6. Build and start all docker services:

#Linux
sudo docker-compose up --build

GitHub - romy47/portfolio at medium_1_docker_dev

What’s Next?

In Part 2, we will purchase a domain, configure the DNS record of that domain, and set up an EC2 instance in AWS.