AWS Lambda + FastAPI (Serverless Deployment): Complete CI/CD Pipeline Using GitHub Actions
Let’s build a complete CI/CD workflow using GitHub Actions, FastAPI, AWS Lambda (Serverless Deployment) and AWS S3.
Are you burdened with mundane boring and repetitive tasks that delays bringing your magical software product to production? As software/devops engineers focused on bringing our code to life, we all face the same problems where we need to perform tasks that are boring, repetitive and sometimes a total waste of time. For instance, one of them is the process of running the unit tests on various environments and then deploying our software to various delivery platform. The processes takes longer than they should and surely makes our day at work inefficient and unproductive. Have you ever wondered:
- How can I automate all these boring repetitive testing and deployment processes?
- Do we have any kind of method that could solve these problems?
Don't worry you’re not alone.
I thought about it all the time, till I realized that there is a simple solution to these problems. It is an amazing method in software development world called Continuous Integration and Continuous Deployment(CI/CD).
CI/CD is one of the best practices in software development as it reduces the repetitive process of unit testing and the deployment of the software.
This practice undeniably will help the developers in efficiently automating all the steps required for running the automated tests on the server. Plus, it can continuously deploy the application to the deployment platform once it has passed the automated tests. And yes, it is fully automated.
The good news is, all of these processes can be achieved by using a new feature on Github called GitHub Actions!
In this article, we’ll demonstrate you a simple walkthrough on building a complete CI/CD workflow of FastAPI using GitHub Actions and we’ll deploy the API to AWS Lambda.
Let’s Dive in!
Table of Contents:
- Create a GitHub Repository
- Clone your repository to local machine
- Setup Virtual Environment
- Install the Required Dependencies
- Run the API in Local Machine
- Run Unit Test in Local Machine
- Update requirements.txt
- Create GitHub Actions Workflow Directory
- The Components of GitHub Actions
- Continuous Integration (CI): Build Automated Test
- Configuring GitHub Secrets, Amazon S3 and AWS Lambda
- Continuous Deployment (CD): Deploy Lambda
- Running the GitHub workflow
1. Create a GitHub Repository
Create a new repository in GitHub for the project.
In this part, assign a relevant repository name and you’ll also need to add a .gitignore
file. You can select the .gitignore
template and since we are using Python (FastAPI), go ahead select the Python .gitignore
template.
2. Clone your repository to your local machine
In order to clone the your repository, you’ll need to have the link of your GitHub repository.
From your repository page on GitHub, click the green button labeled Code, and in the “Clone with HTTPs” section, copy the URL for your repository. Once you have copied the link, you can now clone the project to your local machine.
Open your bash shell and you can change your current working directory to the location that you want to clone your repository. To clone your repository is simple, you can use:
git clone https://github.com/URL-TO-REPO-HERE
Nice! you have successfully cloned your github repository.
3. Setup Virtual Environment
To follow these steps below, you are required to install Python in your machine. If you do not have it install yet, please check this out: Python 3 Installation & Setup Guide — Real Python
Once you have Python 3 installed, create a virtual environment inside your local project directory. Open a terminal on your local machine and run the following commands:
- Install the virtualenv package
You can install the package by using pip.
pip install virtualenv
2. Create the virtual environment
To create a virtual environment, you must specify a path for that. For example to create one in the local directory called venv
, type the following:
virtualenv venv
3. Activate the virtual environment
You can activate the python virtualenv by running the following command:
- Mac OS / Linux
source venv/bin/activate
- Windows
venv\Scripts\activate
4. Install the Required Dependencies
If you want to follow along the walkthrough, this GitHub repository contains the code for this project. Please feel free to use and follow the codes there.
To install all the libraries for the project, make sure you are in the root of the project and run the following command:
pip install -r requirements.txt
5. Run the API in Local Machine
The sample project we created in this walkthrough tutorial is based on FastAPI. FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints. If, you’re interested in learning more about this cool framework, you can read this article written by its awesome author, Sebastián Ramírez. Take it from us, FastAPI 🤩 is the most efficient way of creating api’s in python.
In order to run the FastAPI app, we can use Uvicorn ASGI server to start the app in our terminal. If you downloaded or cloned the code from GitHub (as mentioned in Point 4), you can follow the commands below.
Alright, let’s change our current working directory to app
folder. Then, start our uvicorn server (hypercorn is another alternative):
cd app && uvicorn main:app --reload
Once we run it, we should be able to access the Swagger UI in our browser. Visit the link:
http://localhost:8000/docs
6. Run Unit Tests in Local Machine
In the project, we have few unit tests that will check the endpoints whether they are working fine or not. The unit tests file is inside app/tests
folder named test_main.py
. To know more about why unit tests are important and how to write them, you may follow our other article here.
from fastapi.testclient import TestClient
from main import appclient = TestClient(app)
def test_main_resource():
response_auth = client.get("/")
assert response_auth.status_code == 200
def test_child_resource():
response_auth = client.get("/api/v1/test")
assert response_auth.status_code == 200
To run the unit tests, open the terminal and type:
pytest
Output in console:
Great, we have passed all the unit tests! Please note that the Pytest library is required to run the test.
7. Update requirements.txt
Incase if you have installed any new libraries or packages to the project, its a good practice to update your requirements.txt by typing the following freeze command:
pip freeze > requirements.txt
This is just to ensure we have the latest updated packages inside our requirements.txt
file so that we can avoid complications when running the projects on the server or any other machine.
8. Create GitHub Actions Workflow Directory
To create the CI/CD workflow in GitHub Actions, we need to create a .yml file in our repository. From our application root, create a folder named .github/workflows
that will contain the GitHub action workflows.
Then, create main.yml
(just an example, we can put any name that you like for the .yml file) inside the created folder as this file will contain all the instructions for our automated tests and deployment process to AWS Lambda through our code on GitHub repository. You can use the code below in the terminal as the guide to achieve this process.
cd path/to/root_repo
mkdir .github/workflows
touch .github/workflows/main.yml
Cool! Now we have the directory for GitHub Actions workflow.
9. Understanding the GitHub Actions Workflow
There are six main components in GitHub Actions:
- Workflow — The automated procedure that we add to our repository and can be triggered or scheduled by an event.
- Events — It is a specific activity that will trigger a workflow. For example, an event triggers when someone pushes a commit to a repository or a pull request is created.
- Jobs —Series of steps that execute on the same runner. It can run parallelly or sequentially depends on our workflow objectives.
- Steps — It is an individual task that will run commands in a job.
- Actions — An action is a set of standalone commands that gets executed on the runner and it is combined into steps to create a job.
- Runners — A server that hosted virtual operating systems by GitHub or your own (self-hosted) that can run commands to perform a build process. Hosted runners by GitHub are based on Microsoft Windows, Ubuntu Linux and macOS.
The workflow that we are going to build will consist of two main jobs:
- Continuous Integration (CI)
The CI will run the automated test, package our FastAPI into Lambda and upload the lambda artifact in the GitHub server in order to enable the other jobs (in our case, the Continuous Deployment Job) to use it.
- Continuous Deployment (CD)
This job will only be executed when the CI job is successfully completed. CD job needs to be dependent on the status of the CI build job in order to make sure that we can only deploy the application once we have passed the CI part. Basically, this job will download the lambda artifact that has been uploaded during the CI job and deploy it to AWS Lambda by linking it with AWS S3.
So our workflow will look like this in the .yml file.
name: CI/CD Pipeline
on:
push:
branches: [ main ]
jobs:
continuous-integration:
....
continuous-deployment:
.....
For the detailing part of the steps and commands inside each job, please stay along with me okay!
Alright, let’s get our hands dirty by creating the workflow using GitHub Actions.
10. Continuous Integration (CI): Build Automated Test and Package Lambda
Here is the complete CI workflow in our main.yml file.
name: CI/CD Pipeline
on:
push:
branches: [ main ]
jobs:
continuous-integration:
runs-on: ubuntu-latest
steps: # Step 1
- uses: actions/checkout@v2
# Step 2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.7
architecture: x64
# Step 3
- name: Install Python Virtual ENV
run: pip3 install virtualenv
# Step 4
- name: Setup Virtual env
uses: actions/cache@v2
id: cache-venv
with:
path: venv
key: ${{ runner.os }}-venv-${{ hashFiles('**/requirements*.txt') }}
restore-keys: |
${{ runner.os }}-venv-
# Step 5
- name: Activate and Install Depencies into Virtual env
run: python -m venv venv && source venv/bin/activate &&
pip3 install -r requirements.txt
if: steps.cache-venv.outputs.cache-hit != 'true' # Step 6
- name: Activate venv and Run Test
run: . venv/bin/activate && pytest
# Step 7
- name: Create Zipfile archive of Dependencies
run: |
cd ./venv/lib/python3.7/site-packages
zip -r9 ../../../../api.zip .
# Step 8
- name: Add App to Zip file
run: cd ./app && zip -g ../api.zip -r .
# Step 9
- name: Upload zip file artifact
uses: actions/upload-artifact@v2
with:
name: api
path: api.zip
Let us break it down and have a look at each part of the workflow:
- The name assigned to this workflow is
CI/CD Pipeline
- The workflow will be triggered when commit codes pushed to the
main
branch in the repository. - The job defined in this workflow is
continuous-integration
. - The runner used in the workflow is
ubuntu-latest
(Ubuntu Linux Operating Systems)
These are the sequential series of steps defined in the CI workflow:
- Step 1: Perform
actions/checkout@v2
that will checkout to our repository and downloads it to the runner. - Step 2: Setup python 3.7 by using actions -
actions/setup-python@v2
- Step 3: Install Python Virtual Environment (
virtualenv
) package - Step 4: Caching Dependencies using
actions/cache@v2
. This step will increase the performance of the project workflow that consists of many large dependencies by efficiently reduce the time required for downloading. For further explanation, please read the GitHub Actions Documentations. - Step 5: Activate the virtualenv and install all the dependencies that consist inside
requirements.txt
- Step 6: Run Unit Tests. By the way, we need to activate again the virtualenv before running the test as GitHub Actions doesn’t preserve the environment.
- Step 7: Package our Lambda by zipping all the dependencies in the
venv
site-packages and place it in our root directory. The zip file is namedapi.zip
- Step 8: Add the contents of our app folder into
api.zip
- Step 9: Upload the
api.zip
to GitHub server as an artifact usingactions/upload-artifact@v2
. This will enable the next job to retrieve back the artifact file for the deployment of our lambda package which is ourapi.zip
file.
Phew! We have completed our CI workflow using GitHub Actions. Let’s make some minor changes in our code and try to push our code to the main branch. Make sure you’re using the CI code above in the main.yml
file to test the CI workflow. This is how the result will look like when we have trigger the workflow by our code push event.
You will see this output by going inside the Actions tab in your repository.
Continous Integration (CI) Done! ✔️
Let’s quickly move onto the Continuous Deployment part now. We are almost there!
11. Configuring GitHub Secrets, Amazon S3 and AWS Lambda
Before we create the CD workflow for our project, FIVE things that need to be done:
- Add GitHub Secrets
- Create S3 Bucket
- Create a Lambda Function
- Implement Mangum Handler
- Update Lambda handler
- Add GitHub Secrets
GitHub Secrets is used to store confidential information. In our case, we need to store our AWS_SECRET_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION.
To add the secrets, click on Settings from the repository page, then select secrets from the left menu-list. We will see a button named New Repository Secrets on the top right and click on that Button. This is the example output in GitHub after we have added the secrets.
After we have added the secrets in our GitHub repository, we can now use the secrets variable in our main.yml
workflow file.
2. Create Amazon S3 Bucket
Go to AWS Management Console and log in using our AWS Account. Then, proceed to Amazon S3 in the console and Click on Create bucket.
Just give the bucket any name and click on Create bucket. The diagram below shows how to create the S3 bucket:
Done, you have successfully created the S3 bucket. Let’s create our Lambda function then.
3. Create a Lambda Function
Go to AWS Management Console and log in using our AWS Account. Then, proceed to AWS Lambda in the console and click on the Create function.
Then, choose author from scratch, select python 3.7 as runtime. You will need to choose or create an execution role before creating the function. Once it is done, click on the Create function.
4. Implement Mangum Handler
In order to enable our FastAPI to be deployed as a Lambda function in AWS, we will need to use the Mangum library to wrap our API. Why we need Mangum? It is because:
- Mangum will manage the responses back from the Lambda function to the API Gateway and it acts as an adapter to handle the API Gateway routes requests to our Lambda function.
To apply Mangum in our code, we just need to add the following code:
from mangum import Mangum
...
handler = Mangum(app=app)
In our main.py
, this is how we implement the Mangum which is the Amazon lambda handler:
from fastapi import FastAPI
from mangum import Mangum # <---------- import Mangum library
from api.v1.api import router as api_router
app = FastAPI(title='Serverless Lambda FastAPI')
app.include_router(api_router, prefix="/api/v1")
@app.get("/", tags=["Endpoint Test"])
def main_endpoint_test():
return {"message": "Welcome CI/CD Pipeline with GitHub Actions!"}
handler = Mangum(app=app) # <----------- wrap the API with Mangum
5. Update Lambda Handler
For this part, we need to update the lambda handler in AWS Lambda Runtime settings.
By default we will see the handler is set to lambda_function.lambda_handler
and we need to update this handler value to match with our FastAPI handler.
Under Runtime Settings click on Edit. Then, update the handler to main.handler
(it means that we retrieve the handler
variable which is the Mangum handler inside main.py
)
12. Continuous Deployment (CD): Deploy Lambda
Let’s continue with our GitHub Actions workflow. So, here is the complete CD workflow in our main.yml
file.
...continuous-deployment:
runs-on: ubuntu-latest
needs: [continuous-integration]
if: github.ref == 'refs/heads/main'
steps: # Step 1
- name: Install AWS CLI
uses: unfor19/install-aws-cli-action@v1
with:
version: 1
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} # Step 2
- name: Download Lambda api.zip
uses: actions/download-artifact@v2
with:
name: api# Step 3
- name: Upload to S3
run: aws s3 cp api.zip s3://<YOUR_S3_BUCKET_NAME>/api.zip
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}# Step 4
- name: Deploy new Lambda
run: aws lambda update-function-code --function-name <YOUR_LAMBDA_FUNCTION_NAME> --s3-bucket <YOUR_S3_BUCKET_NAME> --s3-key api.zip
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
Here is the explanation:
- The job defined in this workflow is
continuous-deployment
. - The runner used in the workflow is
ubuntu-latest
(Ubuntu Linux Operating Systems) - This job will only run when the
continuous-integration
build is succeeded. This process can be achieved by using the commandneeds:[continuous-integration]
after the runner has been defined. - Check if current branch is
main
by using the commandif: github.ref == ‘refs/heads/main’
These are the sequential series of steps defined in the CD workflow:
- Step 1: Install AWS CLI in the runner using
unfor19/install-aws-cli-action@v1
- Step 2: Download
api.zip
artifact from GitHub server usingactions/download-artifact@v2
. The artifact name defines in this step must the same as the name used during the step of uploading the artifact. In our case, the artifact name isapi
- Step 3: Upload
api.zip
to our Amazon S3 bucket name that we have defined. - Step 4: Deploy the
api.zip
as uploaded in S3 to our AWS Lambda function name that we have defined.
P/s: For each of the steps that use AWS CLI, we need to include the environment variables of AWS secret keys inside it. The AWS secret keys will be the GitHub Secrets that we store earlier. This is how it can be done in GitHub Actions .yml file:
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
So, this is how the build succeeded result will look like for the CD workflow when we have triggered it by any push event.
Continuous Deployment (CD) workflow is done! ✔️
13. Running the GitHub Actions CI/CD workflow
Here is the complete CI/CD workflow in our main.yml
file:
name: CI/CD Pipeline
on:
push:
branches: [ main ]
jobs:
continuous-integration:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python all python version
uses: actions/setup-python@v2
with:
python-version: 3.7
architecture: x64
- name: Install Python Virtual ENV
run: pip3 install virtualenv
- name: Setup Virtual env
uses: actions/cache@v2
id: cache-venv
with:
path: venv
key: ${{ runner.os }}-venv-${{ hashFiles('**/requirements*.txt') }}
restore-keys: |
${{ runner.os }}-venv-
- name: Activate and Install Depencies into Virtual env
run: python -m venv venv && source venv/bin/activate &&
pip3 install -r requirements.txt
if: steps.cache-venv.outputs.cache-hit != 'true'
# Install all the app dependencies
- name: Install dependencies
run: pip3 install -r requirements.txt
# Build the app and run tests
- name: Build and Run Test
run: . venv/bin/activate && pytest
- name: Create Zipfile archive of Dependencies
run: |
cd ./venv/lib/python3.7/site-packages
zip -r9 ../../../../api.zip .
- name: Add App to Zipfile
run: cd ./app && zip -g ../api.zip -r .
- name: Upload zip file artifact
uses: actions/upload-artifact@v2
with:
name: api
path: api.zip
continuous-deployment:
runs-on: ubuntu-latest
needs: [continuous-integration]
if: github.ref == 'refs/heads/main'
steps:
- name: Install AWS CLI
uses: unfor19/install-aws-cli-action@v1
with:
version: 1
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
- name: Download Lambda api.zip
uses: actions/download-artifact@v2
with:
name: api
- name: Upload to S3
run: aws s3 cp api.zip s3://<YOUR_S3_BUCKET_NAME>/api.zip
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
- name: Deploy new Lambda
run: aws lambda update-function-code --function-name <YOUR_LAMBDA_FUNCTION_NAME> --s3-bucket <YOUR_S3_BUCKET_NAME> --s3-key api.zip
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
Now, let’s push some changes to our main branch to see the result of our CI/CD workflow in GitHub Actions. We should see something like this in our GitHub Actions for the latest build workflow result.
Awesome! All jobs succeeded. All jobs succeeded. This means we have completed the full workflow of our CI/CD Pipeline for FastAPI to AWS Lambda.
This is extremely powerful. We can now deploy our API, bug fixes and any new feature requests to production in minutes rather than days, without any human dependency. Congratulations!
Summary
Implementing a CI/CD Pipeline in our API development projects is incredibly powerful. IT helps us increase our productivity and confidence without having to spend a lot of time handling the tasks and commands to run these mundane processes manually.
In this article, we discussed the complete CI/CD workflow starting from creating a GitHub Repository until building a complete CI/CD workflow using GitHub Actions. Along the way, we also learned about running ASGI APIs, performing unit tests in our local machine, packaging our FastAPI to Lambda using Mangum, and configuring AWS Lambda and AWS S3 Services for our deployment.
We hope this article will help you build your own customized CI/CD Pipelines for your own awesome FastAPI projects.
Peace! ✌️
References
[1] FastAPI + AWS = Robust API by Anna
[2] Simple Serverless FastAPI with AWS Lambda by deadbearcode.
[4] GitHub Actions docs: Build and Testing Python
[5] Handling Continuous Integration And Delivery With GitHub Actions by Aleem Isiaka
[6] Serverless FastAPI CI/CD with CircleCi by deadbearcode.