Deploying a Python Flask application to AWS Lambda With Serverless Framework and CircleCI

Serverless is the new cool kid. Well, might not be “new” new, but considering that still the majority of web applications are monolithic and quite a few (compared to serverless) are micro-services, its still quite a mysterious technology which often either overlooked or overdone. Some companies are migrating some of their more “serverlessish” functions to cloud as serverless functions, and the use cases are becoming more and more evident in the technology business.

This particular post will be not on the basics of serverless concepts nor any of the tools used here. But I will be brushing on the usability of each of these services and how together these can make a match made in heaven.

To start with, let me introduce the tools which I will be using in this tutorial:

• Python 3.6 or above (the best language ever!)
• Pip (duh!)
• Flask 1.1.2
• NodeJS 12.X (For Serverless Framework)
• Serverless Framework
• AWS CLI
• Circle-CI
• AWS Lambda

Let’s get our hands dirty

Well, first you need the Python Flask application itself. So let’s create a very simple Flask app.

First, it’s advisable (or rather necessary) to have the virtual environment set up. So, this assumes that it’s installed. if not just run;

pip install virtualenv

navigate to the project directory.

virtualenv some_env
source some_env/bin/activate

once, the virtual environment is created and you have sourced to it. Install necessary pip libraries in the environment. What you will need essentially is only Flask for this tutorial, but it does not matter what are the other libraries you will be using if you have them in the requirements list file (ie. requirements.txt).

pip install Flask

Then you will have to add your pip libraries (even though it is just Flask above command will install many more dependencies) to a file called requirements.txt.

pip freeze > requirements.txt

Once Flask is installed in the environment and the requirements file is created, add the following code block to a file called app.py (you can name it anything really, ie. cat.py, inflatedpool.py, etc.py ..)

#app.py
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello():
    return 'Hello, World!'
@app.route('/say-my-name')
def hello_world():
    return 'Hello, ' + request.args.get('name')

Well, this is not quite the most complicated and fun Flask application, but this will do for this tutorial as the essence is not the application itself but the DevOps part of it.

The next thing you would need is a way to deploy this to AWS Lambda.

AWS Lambda

AWS Lambda is the serverless “function as a service” model offering from Amazon Web Services. This is the most cemented serverless offering from any cloud provider and surpasses most of the other offerings such as Cloud Functions from Google or Azure Functions from Microsoft. Lambda has several programming languages supported in the platform, including Python, Go, NodeJS, Ruby, C#, and even PowerShell.

There are multiple ways that AWS has given you to create Lambda functions. Such as using the GUI (the web console), using AWS SAM, or using Cloud Formation. However, in this tutorial, we will be using the Serverless Framework (https://www.serverless.com/) which is an open-source project which was been there for quite some time now. The reason for Serverless Framework to be a foremost tool is that despite AWS has given AWS SAM (Serverless Application Model), Serverless Framework can be used to deploy to any cloud provider, not just AWS. Hence, one could say it is platform agnostic.

Anyways, before jumping to Serverless Framework you should have AWS CLI installed and configured in your machine. If you need to know how to do it, check this out (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)

Serverless Framework

The Serverless Framework has been the de facto leader in serverless configurations management and the most used open-source one. The fact that the framework is platform-agnostic has played a major role in getting the reputation. It’s quite easy to install and there are multiple ways you can do so, though npm would be the easiest given that you do have it. Simply have the serverless package installed globally.

npm install -g serverless

once you’ve done this check if this is installed correctly by typing;

serverless

in a terminal.

Now we have to set up the Serverless configuration so we can manually deploy our serverless app in Lambda.

1. First, you need to install the plugins required for your Flask application in the context of Serverless Framework

npm install serverless-wsgi serverless-python-requirements --save-dev

2. Next, you will have to create the Serverless configuration. This is the place where you will be orchestrating the deployment. Typically, it should be written in YAML. (though CloudFormation supports both the JSON and YAML)

# serverless.yml
service: serverless-flask
# These are the plugins you have installed to support WSGI and pip 
# requirements
plugins:
  - serverless-python-requirements
  - serverless-wsgi
custom:
  wsgi:
    # This is the file name and the app name of the application 
    # Where file name is app.py and Flask app name is app
    # ie. app = Flask(__name__) 
    app: app.app
    packRequirements: false
  pythonRequirements:
    dockerizePip: non-linux
provider:
  name: aws
  runtime: python3.6
  stage: dev
  region: us-east-1
functions:
  app:
    handler: wsgi.handler
    events:
      - http: ANY /
      - http: 'ANY {proxy+}'

Based on your app and your preferences you will need to change the following:

• name: Any name you want to give for your serverless function
• app: This can be anything based on your file name and Flask app name (ie. lazycat:catapp)
• runtime: This could be any version of Python in which your application is written in
• region: The AWS region in which your Lambda function needs to get created.

Save this as serverless.yml in the root directory (in this particular scenario it should be in the same level as the app.py is)

Given that you have AWS CLI set up you only need to run one command to deploy this to Lambda.

sls deploy

This will create all the resources in AWS including API Gateway for you. Also, this will create a folder called .serverless in the root, which will contain the application code zipped (without Serverless Framework we have to manually upload this to a Lambda function), CloudFormation template, and all the requirement wheels.

If the deployment is successful you will get an output with the URL to which the Lambda is assigned to. You can simply navigate to that and check your API.

CircleCI

I’ve been using GitLab CI and Jenkins for most of my life and CircleCI was a new thing to me. But found out that they have a very generous free tier account which is more than enough for you to play around with. Circle CI is a SaaS offering of Continous Integration (CI) and Continous Deployment (CD). It will allow you to automate your build/test and deployment pipeline so that you can hook up your git commits to trigger CI/CD jobs.

However, note that for CircleCI you must have your repositories in either GitHub or BitBucket.

Assuming that you have a GitHub account it is quite easy to set up an application there. Following are the steps:

1. Go to https://circleci.com/signup/ and click on either Sign up using GitHub or BitBucket.
2. Enter your GitHub/BitBucket username and password to login to the next window (and do the 2fa if you have that ).
3. Once you authorized Circle CI with GitHub credentials you will get a list of organizations you are in which you will have to select based on which project you need to set up the CI/CD.
4. Once you select that all the projects in that organization will be listed on the dashboard. It will look something like this.

CircleCI Dashboard.

Next, you can either set up the CI/CD pipelines from the GUI wizard there, by selecting a project or you can set up it manually in your repo. Since this will be a serverless project we will be doing the latter.

Let’s go back to the code.

You remember that we have created a virtual environment and created a small Flask endpoint in app.py and set up the manual serverless deployment to AWS Lambda. Now we have to set up Circle CI to run the Serverless script in the CI/CD environment so we can automate the deployment.

1. First, we need to create a folder in the application root called .circleci and go inside the folder

mkdir .circleci && cd .circleci

2. Next, you will need to create the CircleCI configuration. It should be named config.yml (as per the naming convention)

version: 2
jobs:
  deploy:
    working_dir: ~/python-ci
    docker: 
      - image: circleci/python:3.6.1
    steps: 
      - checkout
      - restore_cache:
          keys:
            - dependencies-node-{{ checksum "package.json" }}
            - dependencies-node
      - run:
          name: Install python dependencies
          command: |
            sudo pip install -r requirements.txt
      - run:
          name: Install node and npm
          command: |
            curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
            sudo apt-get install -y nodejs
            node --version && npm -v
      - run: 
          name: Install Serverless CLI and dependencies
          command: | 
            sudo npm i -g serverless
            npm install
      - save_cache:
          paths:
            - node_modules
          key: dependencies-node-{{ checksum "package.json" }}
- run:
          name: Deploy application
          command: sls deploy -v

Now that we have all these we can push the code to the repository. However, even though we have deployed using locally set up AWS credentials, we need to have our deployment environment set up with those as well. To do that you need to go to your project in CircleCI and add them as environment variables.

Where to find project settings

Environment Variables

Now you can push your code to the repository and see the build running in CircleCI.

CircleCI Build

Now that's all you need to do to deploy your Flask application to Lambda using Serverless Framework and CircleCI.

You can check your AWS console to see all the Lambda and API Gateway resources created.

Thanks!!!