Django Admin on AWS Lambda using Zappa and Github Actions

Published in

Engineering@Wizikey

5 min readSep 2, 2021

One of the reasons why Wizikey’s media part is so good is because we use a combination of technology and some really smart people who know this industry really well. The tech helps the team to make better and faster decisions and actions and the people help to make the ‘technology’ better.

One of the core components for this team to work is Django’s Admin console. We use only the Admin part of Django as the rest of the system runs on Nodejs based Graphql server and Next.js based frontend. Will discuss them sometime later. Django has helped us to make tools for the internal teams really fast and easy.

In Wizikey, we are on this endeavor to move ‘everything’ to code. And this time, we did it for Django. We have used Github Actions and AWS lambda. Again, will discuss sometime later why we chose these two.

Zappa:

Zappa is a CLI tool specifically built for deploying and Managing a Python-based Web Framework compatible with WSGI on AWS Lambda and AWS API Gateway. In just a few commands you can build and deploy your whole application on Cloud.

$ pip install zappa
$ zappa init
$ zappa deploy

Your Django application is live on AWS in just a few minutes, isn’t it amazing? Actually, it is not this simple too. Zappa makes the whole process way lot easier but you still need to follow some easy steps. So, read on!

Prerequisites:

1. Python 3, 3.6 recommended

2. Your Django Application build

3. Python Virtual Environment
For Zappa, you must have an active python virtual environment. if don’t know how to create, just follow these steps:

Open the terminal in the working directory and run the following:

# In you Django application root folder
$ pip3 install virtualenv
$ python3 -m virtualenv venv
$ source venv/bin/activate

4. Set up AWS Credentials
To set up AWS access keys locally on your computer, open up your AWS dashboard console to create an IAM user with Administrator access and grab the AWS credentials section and grab the access_key as well as the access_secret_key. And run the following commands in your terminal.

$ pip3 install awscli --upgrade 
$ aws configure
# Now Put your Access Key ID and Access Secret

5. S3 bucket for Static Files for Django static files
Create an S3 bucket on AWS and give the full public access to the bucket, make sure you uncheck BLOCK ALL THE PUBLIC ACCESS

Now, go to the permission section -> navigate to Cross-Origin Resource Sharing (CORS) section. and add the following JSON:

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET"],
    "AllowedOrigins": ["*"],
    "MaxAgeSeconds": 3000
  }
]

Save the Changes.

6. Configuring Django Settings for Handling Static Files
Install Django S3 storage package

pip install django-s3-storage

Add these lines in your settings.py:

import django-s3-storage  .....  S3_BUCKET_NAME = "PUT-THE-NAME-OF-THE-BUCKET-YOU-CREATED-HERE"
STATICFILES_STORAGE = "django_s3_storage.storage.StaticS3Storage"
AWS_S3_BUCKET_NAME_STATIC = S3_BUCKET_NAME 
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % S3_BUCKET_NAME
STATIC_URL = "https://%s/" % AWS_S3_CUSTOM_DOMAIN

Now we are ready for Rock and Roll!

Steps for local deployment

Open up your Terminal in the root directory of your application and activate your virtual environment
Install all the requirements:

$ pip install -r PATH/TO/YOUR/REQUIREMENTS.TXT

Install Zappa

$ pip install zappa

Initialize Zappa

$ zappa init

It will open a CLI prompt to ask some basic setup questions, no need to panic just choose default values as we can easily change those values in future.
After the setup, it will create a `zappa_settings.json` file in your project (make sure it is placed the same where Django’s manage.py exists)
The file will look like this:

{
    "dev": {
        "aws_region": "us-east-1",
        "django_settings": "app_name.settings",
        "project_name": "project_name",
        "runtime": "python3.6",
        "s3_bucket": "s3_bucket_name",
    }
}

Deploy your application
To update your application:

$ zappa update dev 
$ zappa manage dev "collectstatic --noinput" #RUN ONLY ONCE

Boom!! Your Serverless application has been deployed on AWS Lambda and AWS API Gateway.

But the story is not over yet, remember we need to deploy it using Github Actions. So, let’s replicate the above steps there:

Create a Github Workflow
And add the following in it:

name: Zappa Django deployementon:
  push:
    branches:
      - SPECIFY YOUR BRANCH
jobs:
  deploy-django-serverless:
    runs-on: ubuntu-18.04
    strategy:
      matrix:
        python-version: [3.6]    steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}      - name: Setting up AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_KEY }}
          aws-region: us-east-1      - name: Create virtual env
        run: |
          pip install virtualenv
          python -m virtualenv env      - name: Install Requirements
        run: |
          . env/bin/activate
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          deactivate      - name: Update Django application using zappa
        run: |
          . env/bin/activate
          zappa update dev

And, your CI/CD pipeline is done! In production, you should have proper environments and tests in your pipeline, but this is the minimum setup

Issues faced and their solutions:

AWS Lambda size limit:
The main problem with this approach is you can’t just upload anything to AWS lambda has a size limit of 250 MB, which means the total size of the code (your code + modules) should not be larger than 250 MB. And your project can easily cross this limit especially if you are using big modules. E.g. the NumPy module is ~15MB.

Solution:

Use “slim_handler” in zappa_settings.json, this will upload the package in tar.gz format instead of Zip format.
Remove irrelevant modules from your virtual environment as well as from requirements.txt.

pip uninstall #MODULE-YOU-WANT-TO-REMOVE

If you’re using NLTK in your code then it will be too hard to solve its model download problem, as AWS lambda will not allow you to install any packages. To solve this problem 1st you need to find out which model you need to run your code, for example, for tokenization NLTK use the Punkt module which is about 14MB in size.
Now try to find out which model you need from the Punkt module, after finding out just keep that model only and delete another one, now just copy the nltk_data directory from the home directory to your working directory.
Add this in your __init__.py of the main Django admin project directory (the same directory where settings.py exists)

import nltk nltk.data.path.append("./nltk_data")

This may not look like an ideal solution but it worked. Any suggestions are most welcome!

MySQL binary files issue:
After a successful deployment of your code, you will see Zappa return 500/502 error code, this might be because AWS lambda is not able to find out MySQL connector binary files.

Solution:

Just remove any other MySQL module and install PyMySQL.

pip install pymysql

And add following code in the __init__.py of the main Django admin project directory (the same directory where settings.py exists):

import pymysql
pymysql.install_as_MySQLdb()

Still, getting errors? Just check your AWS Cloudwatch for logs, you might miss any module or it can just be an IAM permission issue.

There are other ways too to do this deployment but this one was the fastest and easiest we could do it.

References:

Some helpful articles: