How To deploy Django with Postgres,Celery,Redis, Nginx, and Gunicorn on VPS with Ubuntu 22.04 | 2023 [Best practices]

14 min readOct 1, 2023

introduction

Django is a powerful web framework that can help you get your Python application or website off the ground. Django includes a simplified development server for testing your code locally, but for anything even slightly production related, a more secure and powerful web server is required.

Briefly introduce the topic of deploying a Django web application with a modern tech stack.
Explain the importance of using technologies like Postgres, Celery, Nginx, and Gunicorn for a robust and performant deployment.
Mention that the article will guide readers through the step-by-step process.

Prerequisites and Goals

In order to complete this guide, you should have a fresh Ubuntu 22.04 server instance with a basic firewall and a non-root user with sudo privileges configured. You can learn how to set this up by running through (degitalocean,aws,ovh,…) initial server setup guide.

You will be installing Django within a virtual environment. Installing Django into an environment specific to your project will allow your projects and their requirements to be handled separately.

Once you have your database and application up and running, you will install and configure the Gunicorn application server. This will serve as an interface to our application, translating client requests from HTTP to Python calls that our application can process. You will then set up Nginx in front of Gunicorn to take advantage of its high performance connection handling mechanisms and its easy-to-implement security features.

Let’s get started.

Installing the Packages from the Ubuntu Repositories

To begin the process, you will download and install all of the items that you need from the Ubuntu repositories. Later you will use the Python package manager pip to install additional components.

First you need to update the local apt package index and then download and install the packages. The packages that you install depend on which version of Python your project will use.

If you are using Django with Python 3, type:

sudo apt update
sudo apt install python3-venv python3-dev libpq-dev postgresql postgresql-contrib nginx curl

This command will install a tool to create virtual environments for your Python projects, the Python development files needed to build Gunicorn later, the Postgres database system and the libraries needed to interact with it, and the Nginx web server.

Creating the PostgreSQL Database and User

Now you can jump right in and create a database and database user for our Django application.

By default, Postgres uses an authentication scheme called “peer authentication” for local connections. Basically, this means that if the user’s operating system username matches a valid Postgres username, that user can login with no further authentication.

During the Postgres installation, an operating system user named postgres was created to correspond to the postgres PostgreSQL administrative user. You need to use this user to perform administrative tasks. You can use sudo and pass in the username with the -u option.

Log into an interactive Postgres session by typing:

sudo -u postgres psql

You will be given a PostgreSQL prompt where you can set up our requirements.

First, create a database for your project:

CREATE DATABASE myproject;

Note: Every Postgres statement must end with a semi-colon, so make sure that your command ends with one if you are experiencing issues.

Next, create a database user for our project. Make sure to select a secure password:

CREATE USER myprojectuser WITH PASSWORD 'password';

Afterwards, you’ll modify a few of the connection parameters for the user that you just created. This will speed up database operations so that the correct values do not have to be queried and set each time a connection is established.

You will set the default character encoding to UTF-8, which Django expects. You are also setting the default transaction isolation scheme to “read committed”, which blocks reads from uncommitted transactions. Lastly, you are setting the timezone. By default, Django projects will be set to use UTC. These are all recommendations from the Django project itself:

ALTER ROLE myprojectuser SET client_encoding TO 'utf8';
ALTER ROLE myprojectuser SET default_transaction_isolation TO 'read committed';
ALTER ROLE myprojectuser SET timezone TO 'UTC';

Now, you can give the new user access to administer the new database:

GRANT ALL PRIVILEGES ON DATABASE myproject TO myprojectuser;

\q

Postgres is now set up so that Django can connect to and manage its database information.

Creating a Python Virtual Environment for your Project

Now that you have a database ready, you can begin getting the rest of your project requirements. You will install the Python requirements within a virtual environment for easier management.

First, create and change into a directory where your can keep your project files:

mkdir ~/myprojectdir
cd ~/myprojectdir

Within the project directory, create a Python virtual environment by typing:

python3 -m venv myprojectenv

This will create a directory called myprojectenv within your myprojectdir directory. Inside, it will install a local version of Python and a local version of pip to manage packages. You can use this virtual environment structure to install and configure an isolated Python environment for any project that you want to create.

Before installing your project’s Python requirements, you will need to activate the virtual environment. You can do that by typing:

source myprojectenv/bin/activate

Your prompt should change to indicate that you are now operating within a Python virtual environment. It will look something like this: (myprojectenv)user@host:~/myprojectdir$.

With your virtual environment active, install Django, Gunicorn, and the psycopg2 PostgreSQL adaptor with the local instance of pip:

Note: When the virtual environment is activated (when your prompt has (myprojectenv) preceding it), use pip instead of pip3, even if you are using Python 3. The virtual environment’s copy of the tool is always named pip, regardless of the Python version.

pip install django gunicorn psycopg2-binary

You should now have all of the software needed to start a Django project.

Cloning and Configuring a New Django Project

git clone <your_project_url>
cd your_project

Create a virtualenv folder

virtualenv env

install project requirements

pip install -r requirements.txt

create .env file contains :

DEBUG=False
SECRET_KEY=your django project secretkey
ALLOWED_HOSTS=domain_or_ip_1,domain_or_ip_2,..
DB_NAME=your_db_name
DB_USER=your_db_user
DB_PASSWORD=your_db_password
DB_HOST=localhost

in settings.py

from decouple import config,Csv #pip install python-decouple
...

DEBUG=config("DEBUG",False)
SECRET_KEY=config("SECRET_KEY")
ALLOWED_HOSTS=config("ALLOWED_HOSTS",cast=Csv())
...

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': config("DB_NAME"),
        'USER': config("DB_USER"),
        'PASSWORD': config("DB_PASSWORD"),
        'HOST': config("DB_HOST",'localhost'),
        'PORT': '',
    }
}

Next, move down to the bottom of the file and add a setting indicating where the static files should be placed. This is necessary so that Nginx can handle requests for these items. The following line tells Django to place them in a directory called static in the base project directory:

. . .
STATIC_URL = 'static/'

DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'

import os
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

Save and close the file when you are finished.

Completing Initial Project Setup

Now, you can migrate the initial database schema to our PostgreSQL database using the management script:

python3 manage.py makemigrations
python3 manage.py migrate

Create an administrative user for the project by typing:

python3 manage.py createsuperuser

You will have to select a username, provide an email address, and choose and confirm a password.

You can collect all of the static content into the directory location that you configured by typing:

python3 manage.py collectstatic

You will have to confirm the operation. The static files will then be placed in a directory called static within your project directory.

If you followed the initial server setup guide, you should have a UFW firewall protecting your server. In order to test the development server, you need to allow access to the port you’ll be using.

Create an exception for port 8000 by typing:

sudo ufw allow 8000

Finally, you can test out your project by starting up the Django development server with this command:

python3 manage.py runserver 0.0.0.0:8000

In your web browser, visit your server’s domain name or IP address followed by :8000:

http://server_domain_or_IP:8000

You should receive your Django app index page:

If you append /admin to the end of the URL in the address bar, you will be prompted for the administrative username and password you created with the createsuperuser command:

Testing Gunicorn’s Ability to Serve the Project

The last thing you need to do before leaving your virtual environment is test Gunicorn to make sure that it can serve the application. You can do this by entering the project directory and using gunicorn to load the project’s WSGI module:

cd ~/myprojectdir
gunicorn - bind 0.0.0.0:8000 myproject.wsgi

This will start Gunicorn on the same interface that the Django development server was running on. You can go back and test the app again in your browser.

Note: The admin interface will not have any of the styling applied since Gunicorn does not know how to find the static CSS content responsible for this.

You passed Gunicorn a module by specifying the relative directory path to Django’s wsgi.py file, which is the entry point to your application, using Python’s module syntax. Inside of this file, a function called application is defined, which is used to communicate with the application. To learn more about the WSGI specification, click here.

When you are finished testing, hit CTRL-C in the terminal window to stop Gunicorn.

You’re now finished configuring your Django application. You can back out of our virtual environment by typing:

deactivate

Creating systemd Socket and Service Files for Gunicorn

You have tested that Gunicorn can interact with our Django application, but you should now implement a more robust way of starting and stopping the application server. To accomplish this, you’ll make systemd service and socket files.

The Gunicorn socket will be created at boot and will listen for connections. When a connection occurs, systemd will automatically start the Gunicorn process to handle the connection.

Start by creating and opening a systemd socket file for Gunicorn with sudo privileges:

sudo nano /etc/systemd/system/gunicorn.socket

Inside, you will create a [Unit] section to describe the socket, a [Socket] section to define the socket location, and an [Install] section to make sure the socket is created at the right time:

[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn.sock

[Install]
WantedBy=sockets.target

Save and close the file when you are finished.

Next, create and open a systemd service file for Gunicorn with sudo privileges in your text editor. The service filename should match the socket filename with the exception of the extension:

sudo nano /etc/systemd/system/gunicorn.service

Start with the [Unit] section, which is used to specify metadata and dependencies. Put a description of the service here and tell the init system to only start this after the networking target has been reached. Because your service relies on the socket from the socket file, you need to include a Requires directive to indicate that relationship:

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

Next, you’ll open up the [Service] section. Specify the user and group that you want to process to run under. You will give your regular user account ownership of the process since it owns all of the relevant files. You’ll give group ownership to the www-data group so that Nginx can communicate easily with Gunicorn.

Then you’ll map out the working directory and specify the command to use to start the service. In this case, you have to specify the full path to the Gunicorn executable, which is installed within our virtual environment. You will then bind the process to the Unix socket you created within the /run directory so that the process can communicate with Nginx. You log all data to standard output so that the journald process can collect the Gunicorn logs. You can also specify any optional Gunicorn tweaks here. For example, you specified 3 worker processes in this case:

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=ahmed
Group=www-data
WorkingDirectory=/home/ahmed/myprojectdir
ExecStart=/home/ahmed/myprojectdir/myprojectenv/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/run/gunicorn.sock \
          myproject.wsgi:application

Finally, you’ll add an [Install] section. This will tell systemd what to link this service to if you enable it to start at boot. You want this service to start when the regular multi-user system is up and running:

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=ahmed
Group=www-data
WorkingDirectory=/home/ahmed/myprojectdir
ExecStart=/home/ahmed/myprojectdir/myprojectenv/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/run/gunicorn.sock \
          myproject.wsgi:application

[Install]
WantedBy=multi-user.target

With that, your systemd service file is complete. Save and close it now.

You can now start and enable the Gunicorn socket. This will create the socket file at /run/gunicorn.sock now and at boot. When a connection is made to that socket, systemd will automatically start the gunicorn.service to handle it:

sudo systemctl start gunicorn.socket
sudo systemctl enable gunicorn.socket

You can confirm that the operation was successful by checking for the socket file.

Checking for the Gunicorn Socket File

Check the status of the process to find out whether it was able to start:

sudo systemctl status gunicorn.socket

Output
● gunicorn.socket - gunicorn socket
     Loaded: loaded (/etc/systemd/system/gunicorn.socket; enabled; vendor preset: enabled)
     Active: active (listening) since Mon 2022-04-18 17:53:25 UTC; 5s ago
   Triggers: ● gunicorn.service
     Listen: /run/gunicorn.sock (Stream)
     CGroup: /system.slice/gunicorn.socket

Oct 02 17:53:25 django systemd[1]: Listening on gunicorn socket.

Next, check for the existence of the gunicorn.sock file within the /run directory:

file /run/gunicorn.sock

Output
/run/gunicorn.sock: socket

If the systemctl status command indicated that an error occurred or if you do not find the gunicorn.sock file in the directory, it’s an indication that the Gunicorn socket was not able to be created correctly. Check the Gunicorn socket’s logs by typing:

sudo journalctl -u gunicorn.socket

Testing Socket Activation

Currently, if you’ve only started the gunicorn.socket unit, the gunicorn.service will not be active yet since the socket has not yet received any connections. You can check this by typing:

sudo systemctl status gunicorn

Output
○ gunicorn.service - gunicorn daemon
     Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● gunicorn.socket

sudo systemctl status gunicorn

Output
● gunicorn.service - gunicorn daemon
     Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-10-02 17:54:49 UTC; 5s ago
TriggeredBy: ● gunicorn.socket
   Main PID: 102674 (gunicorn)
      Tasks: 4 (limit: 4665)
     Memory: 94.2M
        CPU: 885ms
     CGroup: /system.slice/gunicorn.service
             ├─102674 /home/ahmed/myprojectdir/myprojectenv/bin/python3 /home/ahmed/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunicorn.sock myproject.wsgi:application
             ├─102675 /home/ahmed/myprojectdir/myprojectenv/bin/python3 /home/ahmed/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunicorn.sock myproject.wsgi:application
             ├─102676 /home/ahmed/myprojectdir/myprojectenv/bin/python3 /home/ahmed/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunicorn.sock myproject.wsgi:application
             └─102677 /home/ahmed/myprojectdir/myprojectenv/bin/python3 /home/ahmed/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunicorn.sock myproject.wsgi:application

Oct 02 17:54:49 django systemd[1]: Started gunicorn daemon.
Apr 02 17:54:49 django gunicorn[102674]: [2023-10-02 17:54:49 +0000] [102674] [INFO] Starting gunicorn 20.1.0
Oct 02 17:54:49 django gunicorn[102674]: [2023-10-02 17:54:49 +0000] [102674] [INFO] Listening at: unix:/run/gunicorn.sock (102674)
Oct 02 17:54:49 django gunicorn[102674]: [2023-10-02 17:54:49 +0000] [102674] [INFO] Using worker: sync
Oct 02 17:54:49 django gunicorn[102675]: [2023-10-02 17:54:49 +0000] [102675] [INFO] Booting worker with pid: 102675
Oct 02 17:54:49 django gunicorn[102676]: [2023-10-02 17:54:49 +0000] [102676] [INFO] Booting worker with pid: 102676
Oct 02 17:54:50 django gunicorn[102677]: [2023-10-02 17:54:50 +0000] [102677] [INFO] Booting worker with pid: 102677

Check your /etc/systemd/system/gunicorn.service file for problems. If you make changes to the /etc/systemd/system/gunicorn.service file, reload the daemon to reread the service definition and restart the Gunicorn process by typing:

sudo systemctl daemon-reload
sudo systemctl restart gunicorn

Configure Nginx to Proxy Pass to Gunicorn

Now that Gunicorn is set up, you need to configure Nginx to pass traffic to the process.

Start by creating and opening a new server block in Nginx’s sites-available directory:

sudo nano /etc/nginx/sites-available/myproject

Inside, open up a new server block. You will start by specifying that this block should listen on the normal port 80 and that it should respond to your server’s domain name or IP address:

server {
    listen 80;
    server_name server_domain_or_IP;
}

Next, you will tell Nginx to ignore any problems with finding a favicon. You will also tell it where to find the static assets that you collected in your ~/myprojectdir/static directory. All of these files have a standard URI prefix of “/static”, so you can create a location block to match those requests:

server {
    listen 80;
    server_name server_domain_or_IP;

    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /home/ahmed/myprojectdir;
    }
}

Finally, create a location / {} block to match all other requests. Inside of this location, you’ll include the standard proxy_params file included with the Nginx installation and then pass the traffic directly to the Gunicorn socket:

server {
    listen 80;
    server_name server_domain_or_IP;

    location = /favicon.ico { access_log off; log_not_found off; }

    location /static/ {
        root /home/ahmed/myprojectdir;
    }

    location /media/ {
            root /home/ahmed/myprojectdir;
        }

    location / {
        include proxy_params;
        proxy_pass http://unix:/run/gunicorn.sock;
    }
}

Save and close the file when you are finished. Now, you can enable the file by linking it to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled

Test your Nginx configuration for syntax errors by typing:

sudo nginx -t

If no errors are reported, go ahead and restart Nginx by typing:

sudo systemctl restart nginx

Finally, you need to open up your firewall to normal traffic on port 80. Since you no longer need access to the development server, you can remove the rule to open port 8000 as well:

sudo ufw delete allow 8000
sudo ufw allow 'Nginx Full'

Note: After configuring Nginx, the next step should be securing traffic to the server using SSL/TLS. This is important because without it, all information, including passwords are sent over the network in plain tex

Troubleshooting Nginx and Gunicorn

When You Setup Full Process Correctly But You See Your Static File Not Serving Properly.That’s Mean Your Website not Showing CSS Or Your website Style Correctly.

Sometimes Nginx Not Serving Static File Correctly, They are Just Permission Denied.
To See Log & Get Your Error.

sudo tail -30 /var/log/nginx/error.log

You should give the user that runs the Nginx process permissions to read the file.

sudo usermod -a -G your-user www-data

Next Test & Restart Your Nginx Server.

sudo nginx -t
sudo systemctl restart nginx

Setup Celery and Celery beat

1 Create the celery configuration file

Now let’s create the celery configuration file inside the /etc/default/celeryd directory,

Note: Please replace the user with your ubuntu user, CELERY_APP_NAME, CELERYD_CHDIR, and CELERY_BIN.
*** settings.py , It must contain the following configurations:

...
# Celery Configuration Options
CELERY_BROKER_URL = f"redis://127.0.0.1:6379"
BROKER_URL = f"redis://127.0.0.1:6379"
CELERY_TIMEZONE = TIME_ZONE
CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_TIME_LIMIT = 30 * 60
CELERY_RESULT_EXTENDED = True
CELERY_worker_state_db = True
CELERY_result_persistent=True
CELERY_RESULT_BACKEND = 'django-db'

...

# Name of nodes to start
# here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/ahmed/myproject/env/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="myproject.celery:app"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=2"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
#   and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"

# you may wish to add these options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"

Now lets change the owner of the celery log and PID file,

chown -R celery:celery /var/log/celery/
chown -R celery:celery /var/run/celery/

2 Create the systemd file

ow lets create the systemd file inside /etc/systemd/system/celery.service directory and write following content,

Note: Please replace the user with your ubuntu user and make sure to check the path to bin celery file (this file will available probably inside your virtual environment)

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=ahmed
Group=www-data
EnvironmentFile=/etc/default/celeryd
WorkingDirectory=/home/ahmed/project/
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
  --pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

3 Restart the server

sudo systemctl daemon-reload
sudo systemctl enable celery
sudo systemctl restart celery

Now your celery is ready to use in production. You can check the status of the celery manually by typing following code,

celery -A CELERY_APP_NAME worker -l INFO

Thanks for reading, good luck .

Need help? contact me .

email : ahmedtouahria2001@gmail.com

linked in : https://www.linkedin.com/in/ahmed-touahria-735185237/