Automating Lambda Layers with SAM and Jenkins

Tomer Levi
Sep 22, 2020 · 5 min read

Our data orchestration platform is using several lambda functions to manage, audit, and monitor data processes.
Our Python lambda functions share some logic such as logging, SQS utilities, slack communication code, and more.

To reuse Python modules we used to copy a bunch of .py files containing our shared modules into multiple lambdas. This, obviously is not a good practice as we had to build, pack, and deploy all lambdas upon changes in shared modules.

Lambda Layers

You can configure your Lambda function to pull in additional code and content in the form of layers. A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package.

In addition:

Layers let you keep your deployment package small, which makes development easier. You can avoid errors that can occur when you install and package dependencies with your function code. For Node.js, Python, and Ruby functions, you can develop your function code in the Lambda console as long as you keep your deployment package under 3 MB.

Creating a Layer

  1. Add requests 2.24.0 into a requirements.txt file
  2. Install the requirements using pip into a destination folder:
    pip install -r requirements.txt -t python/lib/python3.7/site-packages/
  3. Zip that folder as layer.zip
  4. Create a layer in the AWS Lambda console and upload the zip
Creation of Lambda Layer in AWS Lambda console
Creation of Lambda Layer in AWS Lambda console
Creating a Lambda Layer in AWS Lambda Console

Obstacles

  1. Time — even if you’ll use Docker to build the right binaries it will be time-consuming.

Pivoting to AWS SAM CLI

Using SAM CLI you can build any lambda layer (and many other resources such as Lambda functions, API Gateway APIs, and more) without taking care of docker images and runtime issues (Docker is necessary for SAM CLI).

In addition, SAM CLI is using CloudFormation to manage your resources, this will make it easier for you to update your layer (or other AWS resources) or delete it in the future.

How Are We Doing It?

Building Lambda Layers using Jenkins and SAM at Fundbox

Steps:

  • A Jenkins job is used to pull a requrements.txt file from a Github repo.
  • Jenkins runs SAM CLI commands to build, pack and deploy the layer.
  • SAM CLI installs packages from PyPI, zip the packages, upload the zip to S3, and uses a CloudFormation template to deploy the layer to AWS.
  • AWS Lambda service will download the layer code from S3 and create the layer

SAM CLI is using templates (YAML files) to describe the resources in AWS. If you create the basic SAM project using SAM CLI you end up with a template similar to this:

HelloWorldFunction is the resource we manage using SAM CLI, it a Python3.7 method and it has 1 trigger event from API Gateway.

SAM Template

template.yaml

In order to get from a template to a real layer, we need to run a few SAM CLI commands. As can be seen in line #6 and #10 our template requires 2 parameters LayerHumanName and Description, these parameters are used later in the code.

As automation is our goal, we created a simple .sh script that will execute SAM CLI commands for us:

Pay careful attention to line #13 especially to use-container flag:

sam build --use-container ...

From SAM documentation, the use-container option means:

If your functions depend on packages that have natively compiled dependencies, use this flag to build your function inside an AWS Lambda-like Docker container.

In other words, if you will not turn this flag on, you might end up with incompatible lambda binaries. So, turn it on :)

Jenkins

Lambda Layer creation parameters in Jenkins

Code:

Build Lambda Layer Jenkins file

The most important part in this Jenkins job is line #43 which runs our requirements_to_layer.sh:

sh "sh requirements_to_layer.sh ${local_repo_path}/...

However, to enable the smooth execution of this line we first need to have it in Jenkins and use our AWS credentials.

In lines 1–11, we define the UI parameters needed for this job (the ones you saw in the screenshot). Later, in lines 14–19, we run a few assertions to verify the parameters were entered correctly by the user.

All 3 code files — sam-lambda-layer.yaml, requirements_to_lambda.sh and lambda_layers.Jenkins stored in a Github repo. We configured our Jenkins job to use that repo. Line #24, ‘checkout scm’ downloads these files from the repo into Jenkins.

Later, we do this:

withAWS(credentials: ..., region: '<your-aws-region>')

‘withAWS’ is using a Jenkins plugin to inject our AWS credentials into this job. ‘sam build', ‘sam package’ and ‘sam deploy’ commands require AWS credentials.

Line #29 is not mandatory but recommended, using ‘withDockerContainer’ we run our code inside a Docker container that already contains SAM CLI rather than installing SAM CLI on Jenkins.
This is a good practice as it makes it easier to upgrade SAM CLI in the future.

withDockerContainer(..., image: 'lambci/lambda:build-python3.7')

Note: you can also use the official SAM emulation Docker images

Summary

To complete the automation picture, we use Jenkins job that simply runs SAM CLI commands for us.

This blog post was written in collaboration with Boris Churzin

Fundbox Engineering

Technology to grow your business.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store