https://www.pexels.com/photo/rows-of-wooden-panels-of-lumber-construction-4178808/

Automating Lambda Layers with SAM and Jenkins

Our data orchestration platform is using several lambda functions to manage, audit, and monitor data processes.
Our Python lambda functions share some logic such as logging, SQS utilities, slack communication code, and more.

To reuse Python modules we used to copy a bunch of .py files containing our shared modules into multiple lambdas. This, obviously is not a good practice as we had to build, pack, and deploy all lambdas upon changes in shared modules.

Lambda Layers

Instead of duplicating code, we decided to share code between lambdas using Lambda Layers.

You can configure your Lambda function to pull in additional code and content in the form of layers. A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package.

AWS Lambda layers

In addition:

Layers let you keep your deployment package small, which makes development easier. You can avoid errors that can occur when you install and package dependencies with your function code. For Node.js, Python, and Ruby functions, you can develop your function code in the Lambda console as long as you keep your deployment package under 3 MB.

Creating a Layer

Assuming we have a Python3.7 compatible lambda function, and we want to add requests version 2.24.0 package to our lambda inside a layer. We need to do the following:

1. Add requests 2.24.0 into a requirements.txt file
2. Install the requirements using pip into a destination folder:
   pip install -r requirements.txt -t python/lib/python3.7/site-packages/
3. Zip that folder as layer.zip
4. Create a layer in the AWS Lambda console and upload the zip

Creating a Lambda Layer in AWS Lambda Console

Obstacles

1. Binaries — as you probably guessed, your local python binaries might be different from the Lambda runtime.
   In other words, code running on your laptop can sometimes behave differently in lambda function due to runtime differences. Meaning that you need a Lambda Python3.7 runtime when you install your packages.
   To work around this, you can use a local Lambda runtime by running a docker image. Matthew Vielkind wrote a wonderful blog post describing the steps in detail.
2. Time — even if you’ll use Docker to build the right binaries it will be time-consuming.

Pivoting to AWS SAM CLI

This process can be much easier if you’ll use AWS SAM CLI.

Using SAM CLI you can build any lambda layer (and many other resources such as Lambda functions, API Gateway APIs, and more) without taking care of docker images and runtime issues (Docker is necessary for SAM CLI).

In addition, SAM CLI is using CloudFormation to manage your resources, this will make it easier for you to update your layer (or other AWS resources) or delete it in the future.

How Are We Doing It?

Building Lambda Layers using Jenkins and SAM at Fundbox

Steps:

• A Jenkins job is used to pull a requrements.txt file from a Github repo.
• Jenkins runs SAM CLI commands to build, pack and deploy the layer.
• SAM CLI installs packages from PyPI, zip the packages, upload the zip to S3, and uses a CloudFormation template to deploy the layer to AWS.
• AWS Lambda service will download the layer code from S3 and create the layer

SAM CLI is using templates (YAML files) to describe the resources in AWS. If you create the basic SAM project using SAM CLI you end up with a template similar to this:

HelloWorldFunction is the resource we manage using SAM CLI, it a Python3.7 method and it has 1 trigger event from API Gateway.

SAM Template

In our Lambda Layer case, the template is:

template.yaml

In order to get from a template to a real layer, we need to run a few SAM CLI commands. As can be seen in line #6 and #10 our template requires 2 parameters LayerHumanName and Description, these parameters are used later in the code.

As automation is our goal, we created a simple .sh script that will execute SAM CLI commands for us:

Pay careful attention to line #13 especially to use-container flag:

sam build --use-container ...

From SAM documentation, the use-container option means:

If your functions depend on packages that have natively compiled dependencies, use this flag to build your function inside an AWS Lambda-like Docker container.

In other words, if you will not turn this flag on, you might end up with incompatible lambda binaries. So, turn it on :)

Jenkins

To make the process of Lambda Layer creation as simple as possible we formulated a Jenkins job. A user who wants to create a Layer required to fill this simple Jenkins job form:

Lambda Layer creation parameters in Jenkins

Code:

Build Lambda Layer Jenkins file

The most important part in this Jenkins job is line #43 which runs our requirements_to_layer.sh:

sh "sh requirements_to_layer.sh ${local_repo_path}/...

However, to enable the smooth execution of this line we first need to have it in Jenkins and use our AWS credentials.

In lines 1–11, we define the UI parameters needed for this job (the ones you saw in the screenshot). Later, in lines 14–19, we run a few assertions to verify the parameters were entered correctly by the user.

All 3 code files — sam-lambda-layer.yaml, requirements_to_lambda.sh and lambda_layers.Jenkins stored in a Github repo. We configured our Jenkins job to use that repo. Line #24, ‘checkout scm’ downloads these files from the repo into Jenkins.

Later, we do this:

withAWS(credentials: ..., region: '<your-aws-region>')

‘withAWS’ is using a Jenkins plugin to inject our AWS credentials into this job. ‘sam build', ‘sam package’ and ‘sam deploy’ commands require AWS credentials.

Line #29 is not mandatory but recommended, using ‘withDockerContainer’ we run our code inside a Docker container that already contains SAM CLI rather than installing SAM CLI on Jenkins.
This is a good practice as it makes it easier to upgrade SAM CLI in the future.

withDockerContainer(..., image: 'lambci/lambda:build-python3.7')

Note: you can also use the official SAM emulation Docker images

Summary

SAM CLI is simplifying the process of creating Lambda Layers. As an alternative to using manual commands and duplicating shared modules we now use a much easier process.
Since SAM is utilizing CloudFormation, we could also manage versions easily and manage our AWS resources in a neater way.

To complete the automation picture, we use Jenkins job that simply runs SAM CLI commands for us.

This blog post was written in collaboration with Boris Churzin