Ground Up Python Deployments

Deploying a simple web application can be a daunting task if you have never painstakingly done it by hand. There are numerous configuration files, package installations, and concepts involved in getting your brand new app up and running. Deploying your application traverses many different layers of the stack and requires at least basic knowledge of each to successfully deploy to the public internet. Over the last 4 years I have found many tutorials and copy-paste guides which do a great job at getting your app running on a publicly accessible server, but fail at teaching you how each of the technologies relate with one another along the way. This article is the first of many I will be writing as an attempt to organize my own notes as well as set groundwork for a technology presentation I am planning for a local technology conferences.

The Reverse Proxy Server

You will probably see this term thrown around a lot when reading articles about Nginx, Apache, or HAProxy. A reverse proxy is an umbrella term which describes a piece of software that typically implements a number of features though request routing is the primary function. Often web servers implement many of these features as well making them a web server/reverse proxy hybrid, this is the case with Nginx but not necessarily the case with services like HAproxy, Apache TS, or Varnish.

1. Load Balancing: receives inbound requests and routes them across two or more HTTP servers (web servers) to spread the load of the traffic. There are a number of different load balancing mechanisms that a given load balancer may implement. (round robin, least-connected, ip-hash), here is a link to fairly definitive list.
2. Caching: stores content from the web servers to reduce the load by returning static content back to the client without having to route the request to the web-server to find the file.
3. Security: acts as a layer of defense in front of the web servers themselves, this could simply mean disallowing direct communication with the server by acting as a proxy or by actually reviewing inbound requests and scanning for malicious code.
4. SSL Acceleration: when SSL is used, the reverse proxy can be the point of termination so the web servers do not take on the extra load of dealing with the encrypted requests.

The reverse proxy server is the first layer to act upon an HTTP request that we will be touching. Imagine it as the front desk at DMV (sad analogy I know… but bear with me). If you walk into the DMV and are looking for information regarding a motorcycle license, the clerk at the desk is going to hand you a small pamphlet (static content) and send you on your way. The clerk does this to keep the DMV from wasting time; just one person with a simple request can eat up a lot of time with desk clerk if the request could have been easily handled up front. A proxy servers job is very much the same once the request is received, the web server looks at the HTTP request (meta, uri, etc.) information and determines if it can handle the request itself or if the request needs to be re-routed (“proxied”) somewhere else, this could be to a web/application server or any other number of services you may be using.

Normally to setup these rules to where the request should be routed, you define URL patterns which when matched against, the reverse proxy sends the request to the designated IP address and port. The proxy can also be configured to check for local static files and serve them if found. This will hopefully make more sense in the tutorial section.

https://www.scalescale.com/tips/nginx/nginx-vs-varnish/

The Application Server

In order for certain requests to be properly fulfilled dynamic actions will need to be taken in order to produce the response. Examples of this would be logging into a your application, checking your Facebook timeline, or submitting a new story on Medium. The reverse proxy/web server itself is unable to run your scripts or connect to your databases, this is a job for your application server. Once the reverse proxy determines (based off the HTTP request it received) that it, by itself, cannot simply send back simple image or css file (static files) but instead needs help from a “specialist”; it then sends the request downstream to the appropriate server based off the proxy configuration you created. This would be DMV case where the front desk gives you a shitty numbered ticket and asks you to wait until the number has been called. The front desk clerk cant handle your advanced “i lost my drivers license” issue and needs to send it to someone who has the training and resources to help you. The application server’s job is to run our code in order to respond to the user with a dynamically generated response.

In order for the application server to properly interface with your application they need to speak a common gateway protocol. As you probably already know, protocols are a set of standardized guidelines for implementing networking communications between servers and/or services. Django, our application framework of choice, implements WSGI (Web Server Gateway Interface) which was designed for python applications. In order for our application to work, we will need both the application as well as the application server to implement the sample protocol. Said in few words, WSGI is the universal interface between applications and the servers which run them.

The Database Server

One can assume that almost all modern applications will use some form of a database solution. MySQL, PostgreSQL, SQLite, MongoDB… the list goes on and on and on and on. Depending on your needs, the database could be installed and run on a separate server to make the infrastructure as modular as possible. In many situations separate databases servers is simply overkill and server can be run from the same machine as your application, which we will be doing later.

The Plan

So enough chit-chat. Lets move on to actually creating a VM and deploying a vanilla Django application. I am going to be very specific with this step by step. I will do my best to explain my rationale for everything I do. The goal of this tutorial is to end with a live vanilla Django deployment along with a broad understanding of the different technology’s responsibilities and how they interact with one another.

For this deployment we will be using the following technologies.

• Hosting Provider: Linode (1024)
• Operating System: Ubuntu Server 16.04
• Reverse Proxy / Web Server: Nginx
• Application Server: Gunicorn
• Language: Python3
• Python Package Manager: PIP
• PIP Virtual Environments: virtualenv
• WSGI Framework: Django 2.0
• Database: PostgreSQL

Setup Linode Instance

If you don’t already have a Linode account, create one. For this example we will be using the lowest tier server instance since I will only be using it for the purpose of this tutorial. Depending on the requirements of your application you may of course need a much more capable machine.

Next, we will need to deploy an operating system image to our new instance. As stated above, we will be using Ubuntu 16.04 LTS.

Once you have selected Deploy an Image and selected Ubuntu 16.04 LTS, it will ask you to supply a master password for your instance. Please, make this a strong password as I have some Russian assshole hack me in the past. After your done deploying the image, you should see this screen. You now have a Powered Off Ubuntu VM ready for use. So lets get started and hit that boot button.Nginx

Once its up, click the Remote Access tab which will give us the SSH command to access our instance. Copy the command into your terminal and you should see this screen.

Once you type yes to continue the connection it will as you for the password you entered when you Deployed the Ubuntu image.

And we’re in.

Create Deployment / Application User

So, when you first access your Ubuntu instance you will be logging in as the root user, the root user has ultimate authority and access within the operating system. When deploying an application, it is wise to silo the files and configurations under a new “application” user which has limited permissions. In this case we will name our new user “djangotutorial”.

To create a new user we user the useradd command with the -m flag, the flag will create a home directory along with the linux user. We will use the home directory to store all of our application files.

At this point, it is important to realize that the djangotutorial user is not part of the sudo group, and therefore cannot install any packages, we will need to do that as the root user. To switch back to the root user, type logout.

Update Apt Repository

Before installing any packages it is important that our apt repository has the newest list of software versions and available updates. When you install a package apt does not automatically check if there is a newer version than the one it already has on file, we need to manually tell it to check.

apt-get update

Install Nginx

Now that we have the most up to date lists of available repositories and their dependencies, lets install our reverse proxy server, Nginx. This is the first gate that our HTTP request will be hitting when someone makes a request to our server. It is the job of the reverse proxy to either serve static files, (images, css, javascript, etc), or route the request to our application server for further processing. All of the routing we desire is done through a configuration file which we will be creating a little later.

apt-get install nginx

At this point we can actually test our reverse proxy server to ensure that it is working properly. Take the ip address of your Linode server that can be found on the “remote access” tab of your Linode account and paste that into your web browser.

TADAAA! As you can see Nginx is response to our HTTP request with its default response. The configuration that Nginx comes with out of the box serves a static HTML web page to any HTTP request made to the base url (ASKJDAHSKJDAS).

The configuration files we will be changing later down the road are located in /etc/nginx/. The default location of the Nginx configuration files could change if you chose a different operating system or use a different package manager than I did in this tutorial.

For now we can take a look at the current configuration which is serving us this default page.

The first file we will look at is the /etc/nginx/nginx.conf, this is the default configuration file that Nginx will load into memory when the service is started, it uses all these configurations to determine what it should do with an incoming request.

As you can see in the red rectangle above, this default configuration file includes all the configuration files found in the /sites-enabled/ and /conf.d/ folders. When the Nginx configuration is loaded, the contents of each of the files found within those folders is dynamically “copied and included” right where that include command is located, creating one massive configuration file. This method allows your configurations to be broken up into smaller units for organizational reasons.

Most everything in the nginx.conf configuration file are default Nginx settings that we wont be worried about within the context of this tutorial, we are interested in the individual server configuration files which are found within the included folders. Since this is a clean install of Nginx, there is only one configuration file inside /sites-enabled/ and /conf.d/ should be empty.

If we run ls -la in the /sites-enabled/ folder we’ll notice that there is one file called default, and it happens to be a symlink back to a file in the /sites-available/ folder. Nginx has the configuration files organized like this to allow you to keep all of the configurations you have ever created in your /sites-available/ folder and the only activate the ones you want by adding a symlink to them in the /sites-enabled/ folder. If you delete the symlink, you will prevent that configuration from being loaded when Nginx starts but you maintain the configuration file.

Now, lets take a look at the symlinked file.

/etc/nginx/sites-available/default

So, I know there is a lot of stuff going on in this file and it does and should look complicated. The reality is, most of what you see here is commented out boilerplate configurations that aren’t even being used. I am going to navigate the source file of the symlink and make a copy of this file.

I will then remove every line that starts with a # so we can see what is actually happening it its most simple form.

WHOA! Look how much better that looks, not so scary after all! Remember, this is just a copy of the default configuration, so this what the server is currently running to display that Nginx splash page we saw earlier.

Lets pick it apart, line by line.

First, every configuration will go into the obligatory server { } structure.

The first two lines simply tell Nginx to accept requests coming in on port 80.

The third line specifies the root directory for requests. This will make sense momentarily.

The fourth line sets which file should be treated as the index if no file is specified within the incoming HTTP requests’s uri. The fifth line sets the server name that Nginx will attempt to mach to the whats found in the incoming URL, by default Nginx sets it to its special “catch all” symbol which will match anything. Since Nginx unaware of your servers IP or your intended domain name at the time of the install, it is set to the wildcard so the default splash page can be displayed as soon as Nginx is installed.

So! As we experienced, if we put our server’s ip into our web browser we are shown a Nginx splash page.

Lets explore why that happened...

When the HTTP request arrives at our server, Nginx attempts to match the URI to ANY of the location configurations that have been loaded from the server configuration files.

http://www.example.com/this/is/an/example/uri/

(anything after the domain in the url is the uri, in our case the domain is the server ip address)

So if we take a look at our example all we have is the ip address as the domain, but there is no uri. If there is no uri, imagine there is an invisible / at the end of the domain / ip address. Notice if you add a slash to the end of the domain and hit enter, Chome will just remove it.

When we hit enter, the browser makes an HTTP request to our server (all HTTP requests are on port 80), and Nginx is listening. It then parses the URI from the domain and since there is “nothing” it resolves to a “/”.

Nginx then attempts to match it to any of the location directives, the only one we have is

Since the URI that was parsed by Nginx is “/” and the location directive is “/” its a match and runs the directives within the curly braces.

Notice, if we add another slash to the URI we will get a 404 response because there was no location match.

In this case its going to try_files, which according to the Nginx documentation.

try_files

Checks the existence of files in the specified order and uses the first found file for request processing; the processing is performed in the current context. The path to a file is constructed from the file parameter according to the root and alias directives. It is possible to check directory’s existence by specifying a slash at the end of a name, e.g. “$uri/”. If none of the files were found, an internal redirect to the uri specified in the last parameter is made

So since, we are requesting the “/” or “index” uri the location matches and is going to search for the index file we specified at the root location.

So it would attempt to find the following files.

/var/www/html/index.html

/var/www/html/index.htm

/var/www/html/index.nginx-debian.html

Lets see if any of them exist.

Look! There is a file. Lets ensure this is the file that Nginx was giving us back by default.

If we were to do request the http://45.56.90.151/test.html, Nginx would attempt to find the test.html file in the /var/www/html/test.htmlLooks like the right one! Lets make a change and refresh our browser to be 100% sure.

VOILA!

So, its all find and dandy that we have found the files that Nginx is setup to serve by default. But, what if we wanted to serve our own custom static HTML file?

Well, because our location directive is setup to try_files (search for and return files if found), and the index URI “/” is mapped to “/var/www/html/” ANY file we put into that directory should be served.

For example, If we were to do request the http://45.56.90.151/test.html, Nginx would match on the “/” attempt to find the test.html file in the /var/www/html/ directory.

Lets try touching a test.html file in the /var/www/html/ directory and test it out.

Now I’ll add some trash to the file.

Lets, change our URL and check it out.

Now that we understand how the current configuration is working, we are ready to explore how Nginx can not only serve static content, but route requests to other applications such as our Application Server if the URI matches what we expect. Since we don’t have an application or an application server setup yet, we will have to get them running before we can move any further with our nginx configuration.

Up to now all we have done.

1. created Linode server
2. deployed Ubuntu image
3. created a user for our application
4. updated our apt repository
5. installed nginx
6. created a copy of the default configuration file /etc/nginx/sites-available/default called djangotutorial

Installing Virtualenv

Since we will be using Django, we are going to need to install it using PIP (the package manager for the Python Package Index or PyPI). When working with application dependencies of any kind it is important to separate them from the packages you have installed on the system level. Luckily there is a beautiful python virtual environment manager called virtualenv.

virtualenv will essentially install your pip packages for your application into a separate folder from your system level pip packages. This is more complex in reality, but for now lets just say it keeps your python dependencies in a separate folder. This way you can have multiple versions of a given package installed on the same server if you are running multiple applications with different package version requirements.

To get stared, lets install pip via the apt package manager. (I know, were installing a package manager with another package manager, fuckin weird)

Since we will be using python3 we need to install python3-pip.

apt-get install python3-pip

pip has A LOT of dependencies so expect it to take a moment.

Now, using our newly installed pip3, lets install virtualenv.

pip install virtualenv

Since for some reason apt didn't give use the latest pip version, lets install it. Pip uses itself to update.

pip install — upgrade pip

Before the update we were required to use the pip3 command after the update we are now about to just use the shorter pip to install the packages we want. As you can see on the last line of the screenshot above, the pip command maps to python3.5.

Creating a Virtualenv

Now that we have virtualenv installed, we are ready to create our first virtual environment to house all of our python dependencies.

First, lets switch back to our application user and navigate to the home directory so we can keep all our files together.

Now, lets create a virtual environment. When we create an virtualenv, a new folder is made in whichever directory we run the command. The folder structure replicates an entire isolated python installation, and creates a python symlink to one of the versions of python existing on our system.

By default it will use whichever python version is symlinked to the default python command. So, in our case its python2.7.

Since we want to use python3 for our project we will need to specify the location of the python executable at the time of creating the virtualenv.

Lets find out where the python3 executable is.

Now, create the virtualenv with the proper python3 path.

TaDa! We have a new folder in this directory with a python installation.

You’ll notice, in the /bin folder. There is an activate file, if we use the source command on this file our virtualenv will be activated and our system will use this python installation rather than the one that exists at the system level. You can tell which virtualenv you are using by looking in front of your username for the (virtualenvname).

Now that we are in our new virtual environment all of the pip installs will go into this folder structure rather than our system’s. I am going to switch back to the application home user folder just to stay organized.

We are now ready to start a bare bones Django project and get her running.

Creating and Configuring Django Project

Lets install Django into our new virtualenv!

pip install Django

And now well start a new project within our application user’s home directory.

django-admin.py startproject djangotutorialproject

We had to name it djangotutorialproject since our virtualenvironment already created a folder with the name djangotutorial.

Lets try to run the test server to see where we are at.

cd djangotutorialproject

python manage.py runserver

Now, refresh your web browser window with the test server running.

SAD! Nothing happened…. why? Because Nginx is not currently configured to pass requests onto the Django test server which is running on http://127.0.0.1:8000/. Lets see if we can connect them.

Configure Nginx

Lets keep our server running in that terminal window and start a new ssh session to do our Nginx modifications, this way we can continuously test our configurations without needing to navigate between users and folders.

Navigate back to /etc/nginx/sites-available/ and open the djangotutorial configuration file we previously made.

Currently, we are only having Nginx serve static files from /var/www/html if the URI matches “/”.

Old Configuration

instead we are going to pass on the request to our local Django server to handle.

New Configuration

Save the file, and navigate to the /etc/nginx/sites-enabled/ folder

lets remove the default symlink and create a new one for our configuration.

now, lets reload our Nginx server to it will find our new configuration.

service nginx reload

Now, refresh your web browser page!

OH NO!! WE GOT AN ERROR.

BUT WAIT ITS A DJANGO ERROR! OUR REQUEST WAS SUCCESSFULLY PASSED TO DJANGO FROM NGINX.

To quickly fix this, lets modify our Django settings file to allow this ip address. Im going to temporarily shutdown my server and make the change.

I added the IP address of my server to the ALLOWED_HOSTS settings variable in my ~/djangotutorialproject/djangotutorialproject/settings.py file.

Original File

After I added to ALLOWED_HOSTS

Save the file and restart the server.

We can then refresh our browser.

HELLL YEAHHHH!

We are now running a vanilla Django2.0 (wsgi application) using the Django TEST server (application server) through a Nginx (webserver/reverse proxy) with python3 and virtualenv.

Don’t get too excited… the Django TEST application server is NOT meant to be used in production and should not be used for anything but testing.

The job of the application server is to run python instances of your Django application on your local operating system. Without an application server, your Django application would never be able to run as a process on your local operating system. The code would just sit there… like a douche bag.

Remember, Django itself and the application server that runs it, are two separate entities. Even though there is a handy application server packaged with Django, that does not mean they are the same thing.

The next step will be installing and configuring a much more capable application server called Gunicorn.

Installing Gunicorn

Luckily for us, Gunicorn is written in Python and can be installed via PIP. Ensure you are still in your virtual environment, and pip install Gunicorn as shown below.

At this point, we can acutally test out our server and attempt to run our vanilla Django application using Gunicorn rather than using the Django Test Application Server.

To do this we need to be in our virtualenv since we installed gunicorn using pip. If we are in our virtualenv the “gunicorn” command should be available.

We need to specify the number of worker processes that we want to spin up of our Django application, I will be using 3 (1 would work for this example but may as well see workers being used). Gunicorn has good documentation explaining how to calculate the number of workers you should use given the hardware setup you have.

Do start up our application server we need give the workers number with the “-w” flag as well as the python module path to our wsgi file that is auto-generated as part of our Django application. It will be sitting in your main app folder within your project folder. In our case it would be

/home/djangotutorial/djangotutorialproject/djangotutorialproject/wsgi.py

(home)/(user-home)/(django-project-folder)/(django-app-folder)/(wsgi)

Python only recognizes folders as modules if there is an __init__.py file in the folder, __init__.py files are normally empty. If you notice our django-project-folder (asdasdsadasd) does not have one, only the django-app-folders are considered python modules. Therefore we need to be in the django-project-directory in order for the command to work.

cd djangotutorialproject

gunicorn -w 3 djangotutorialproject.wsgi

As you can see, 3 worker processes boot up with their own process ids (PIDS). Each of these are an individual instance of your Django application running as a process on your Linode server. They are awaiting HTTP requests to be forwarded from your Nginx server. Once Gunicorn receives the request it will execute the targeted python code within your Django application.

As you can see from the screenshot above the default port for gunicorn is 8000 just like our shitty django development server; this means we shouldnt have to change our nginx config for this to work.

If we refresh our web page you should see our server is up and running.

Daemonizing Gunicorn

So, now that we have Gunicorn setup and tested, it is time that we have it run in the background so we can exit the ssh and not have to worry about killing our server processes. To autostart / monitor Gunicorn we will be using a program called Supervisor.

First, use pip to install supervisor.

pip install supervisor

Once supervisor is installed we can generate a sample configuration file for supervisor that will have the boilerplate code necessary to get started.

echo_supervisord_conf > /etc/supervisor.conf

Lets navigate back to the /home/djangotutorial/ folder and touch a new file called djangotutorial.conf and put in the following configuration. We will go over it line by line.

1. Your name for the process you are creating.
2. The directory for supervisor to change directory to before executing the command.
3. Since we are using gunicorn as our application server, we will point the command option to the gunicorn executable found in our djangotutorial venv
4. the user we want the process to execute under, we will use the user we created
5. the name and location of the logfile we would like the output of the command to be stored in.
6. This redirects the commands stderr output to the stdout file descriptor instead, so that all the logging goes the the logfile described in stdout_logfile rather then having to handle it separately.
7. The signal used to kill the program when a stop is requested. This can be any of TERM, HUP, INT, QUIT, KILL, USR1, or USR2.
8. The number of seconds to wait for the OS to return a SIGCHLD to supervisord after the program has been sent a stopsignal. If this number of seconds elapses before supervisord receives a SIGCHLD from the process, supervisord will attempt to kill it with a final SIGKILL.
9. If true, the flag causes supervisor to send the stop signal to the whole process group and implies killasgroup is true. This is useful for programs, such as Flask in debug mode, that do not propagate stop signals to their children, leaving them orphaned.
10. If true, when resorting to send SIGKILL to the program to terminate it send it to its whole process group instead, taking care of its children as well, useful e.g with Python programs using
11. environment allows us to describe environemnt variables that we would like exported before the command is run. This is VERY important since our command references the wsgi file via python module syntax ie. djangotutorialproject.wsgi. Since we are exporting the PYTHONPATH as /home/djangotutorial/djangotutorialproject/ we can get the wsgi.py located at /home/djangotutorial/djangotutorialproject/djangotutorialproject/wsgi.py

The last thing we need to do before testing our supervisor config is to include our configuration file path in the supervisor.conf file.

vi /etc/supervisor.conf

If you look at the bottom of the file you will find an [include] section, we need to uncomment the default code and include our config file location instead.

files = /home/djangotutorial/*.conf

Starting Supervisor

We can then use supervisorctl to check the status of our djangotutorial process.

We can then open our we browser back up and navigate to our ip address. You should see the Django welcome page! Well done!

Starting Supervisord on System Boot

At this point we have a supervisor configuration to automatically start and restart our application if the server resets or if our Django server fails.

What we don’t yet have is a way to make sure supervisor itself starts up when the server reboots; to do this we will use systemd.

systemd → starts supervisord → starts djangotutorial process

First, create a new file called supervsiord.service in the /etc/systemd/system/ folder.

touch /etc/systemd/system/supervisord.service

Then open the file with your favorite editor and add the following configuration to the file.

[Unit]
Description=Supervisor daemon
Documentation=http://supervisord.org
After=network.target

[Service]
ExecStart=/usr/local/bin/supervisord -n -c /etc/supervisord.conf
ExecStop=/usr/local/bin/supervisorctl $OPTIONS shutdown
ExecReload=/usr/local/bin/supervisorctl $OPTIONS reload
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target
Alias=supervisord.service

Start the systemd process

From now on supervisord will be started when the system starts but since we dont want to reboot the machine we can just start the service manually from the command line using systemctl start.

systemctl start supervisord.service

We can then check our service to make sure its running using

systemctl status supervisord.service

Conclusion

At this point you have an empty Django project deployed behind nginx and gunicorn.

This tutorial is not meant to get you to a production ready deployment but instead to show you how the each service is connected to one another. I hope this was helpful to someone wondering how all this comes together. I would love any feedback, critical or otherwise. Thanks!