Using Custom Lambda Layers to Reduce Code Duplication in AWS Lambda
One of the most well-known principles in the field of software development is “Don’t Repeat Yourself”. Commonly referred to using the “DRY” acronym, its message is quite simple: avoid duplication within a system at all costs.
The DRY principle was introduced by Andy Hunt and Dave Thomas in their 1999 book titled ’The Pragmatic Programmer’. Within, the authors state:
“Every piece of knowledge must have a single, unambiguous, authoritative representation within a system”
While it was the authors' belief that DRY should apply to everything ranging from database schemas and application code to documentation, the question of how strictly it should be followed is hotly debated within the software development community. Oftentimes, the answer differs on a case-by-case basis depending on where logical abstractions make sense for the greatest good of the system. It’s not my goal within this post to try and answer this question or attempt to sway you one way or another. Instead, I’d like to discuss the application of the DRY principle to AWS Lambda Functions through the effective use of Lambda Layers.
If you’ve been involved in the development of code at any level, then you’ll likely agree that having multiple instances of code that performs the same functionality can be burdensome. Duplicate code can affect the overall maintainability of a system by introducing a requirement to apply bug fixes or code changes to multiple places. If these duplicate locations are not managed and identified effectively, this can be a logistical nightmare.
Born long before the advent of both AWS and serverless computing, the DRY principle applies just as equally to code developed for AWS Lambda functions. Whether you subscribe to the theory of having many, single-purpose functions or fewer, monolithic style functions (read this article from Yan Cui for an interesting discussion), removing duplicate code both within functions and across all of your functions should be one of your top priorities.
A recent client engagement saw me working on a solution containing many distinct AWS Lambda functions which all required the ability to communicate with centralized infrastructure component types, including DynamoDB tables, SNS topics and many more. Writing identical code within each Lambda function to interact with the same resource types would have gone against the DRY principle. Instead, I leveraged AWS Lambda Layers to create libraries to be used by all Lambda functions within the solution.
AWS Lambda Layers Explained
As consumers of serverless technology have continued to develop increasingly complex applications, AWS realized the importance of providing the capability to share code between Lambda functions. This was made possible by the release of AWS Lambda Layers, which enables the creation of versioned ZIP packages containing libraries and other dependencies. The libraries and other dependencies packaged within these versioned ZIP files can be used by any Lambda function which adds a specific version of that layer.
The advantages of using Lambda Layers with your Lambda functions are numerous, including:
- Your Lambda function code and deployment package are smaller as a result of having removed dependencies directly from it
- You can tightly control the version of dependencies which you import and use within your Lambda functions without the requirement to package them with each individual function
- You can utilize Lambda Layers published by others within your AWS Organization, or even trusted third-party publishers in the Security, Monitoring and Application Management domains
- Use of published Lambda Layers can be controlled via layer permission policies
Let’s take a look at a couple of examples of dependency management and library creation using Lambda Layers. I’m using Python 3.8 for my examples, so check the AWS Lambda Development Guide to understand the necessary approach for packaging / importing dependencies if using a different Lambda runtime.
Custom Lambda Layer Example 1 — Packaging the Boto3 (AWS SDK) Module
The Boto module (boto3) is an essential part of creating AWS Lambda functions for the Python runtime as it supplies the AWS SDK. The AWS Lambda Python runtime comes packaged with boto3, however, I highly recommend against using it due to an inability to control the exact version of the module being used. This article by Tom McLaughlin is an interesting read into the boto3 and botocore modules provided by the Python Lambda runtime and common misconceptions of them being up-to-date.
Reasoning aside, rather than using the boto3 module provided by the Lambda runtime we can package it into a Lambda layer instead.
Creation of Directory Structure
As outlined within AWS’s documentation on Lambda Layers, a local directory structure should be created for your specific Lambda runtime. For Python, I first create the directory path at python/lib/python3.8/site-packages.
Installation of the boto3 Module using pip
Next, use pip to install the boto3 module (specifying a version if desired) into the newly formed directory structure:
pip install --target python/lib/python3.8/site-packages/ ‘boto3==1.12.14’Additional packages can also be installed into the directory, but we will keep our layer to containing boto3 in this instance.
Packaging of the boto3 Lambda Layer
To package your Lambda layer, use a recursive zip command to archive the contents of the python directory. Within the directory containing your “python” directory, run:
On Linux/macOS:
zip -r lambda_layer_python_boto3.zip ./pythonWindows:
AWS recommends using a zip tool when creating Lambda Deployment Package Zip files in a Windows environment. I tested my example in Windows using 7-Zip to create a Lambda Layer archive with the ‘python’ directory at the root.
Pushing your Lambda Layer to AWS
There are a number of ways to manage the deployment of Lambda Layer ZIP archives to your AWS account (SDK’s, Terraform, CloudFormation etc.). For simplicity, I use the following AWS CLI command:
aws lambda publish-layer-version --layer-name lambda_layer_boto3 --description “Boto3 v1.12.14” --license-info “MIT” --zip-file fileb://lambda_layer_python_boto3.zip --compatible-runtimes python3.8Setting Layer Version Permissions
To control who can call use the new version of the Lambda layer, you can apply a layer version policy. For example, the below command creates a Lambda version policy that ensures that only Lambda’s in AWS account 123456789 can utilize it:
aws lambda add-layer-version-permission --layer-name lambda_layer_boto3 --statement-id account-permission --version-number 1 --action lambda:GetLayerVersion --principal 123456789 --output textAdding the Lambda Layer to a Lambda Function
As with the upload of the layer version, there are many ways to associate a Lambda Layer version with an AWS Lambda function. Below I use the AWS Console below to illustrate the simple process:
- With your AWS Lambda function open, click on the ‘Layers’ box just below the function name:
- In the box at the bottom, select the ‘Add a layer’ button
- Using the dropdown boxes, select the Name and Version of the Lambda Layer you wish to add
- Select the ‘Add’ button and save changes to your function
With the Lambda Layer associated to your function, the boto3 module imported by your function will come from the Lambda layer.
Custom Lambda Layer Example 2 — Custom Library Functions
Example 1 (see above) demonstrated how Lambda Layers can be used to manage and import versioned package dependencies. A more interesting use case, this example will highlight how to package custom libraries within Lambda Layers and use them within your Lambda functions.
A Fictional Scenario
Let’s say that we have several Lambda functions within a central hub AWS account that require access to spoke AWS accounts. A common activity in allowing this to occur would be for these centralized Lambda’s to assume roles in the spoke account via the creation of a Security Token Service (STS) session.
Instead of writing functions within each of the centralized Lambda’s to create STS sessions to our spoke accounts, we will create a Lambda Layer containing a library called aws_helpers which contains a Python function that creates and returns a session object.
The ‘aws_helpers’ Library Layer
Our example library in aws_helper.py contains a single function, create_spoke_account_session, which takes two arguments:
- The spoke account number
- A client connection to STS initialized in the calling Lambda function
The aws_helper.py file is saved within the ./python directory in your structure. You’ll also notice that the function relies on the use of boto3. As was done in Example 1, we can use pip to install the boto3 module within the correct directory structure in our layer to self-contain all dependencies within the layer.
Following creation of the aws_helper.py file and installation of the boto3 module via pip, the directory structure looks like the following:
Zipping and Uploading the aws_helpers Layer
As was the case in Example 1, we package the Lambda layer using a recursive zip command and push to AWS using the ‘aws lambda publish-layer-version’ AWS CLI command:
On Linux/macOS:
zip -r lambda_layer_python_aws_helper.zip ./pythonaws lambda publish-layer-version --layer-name lambda_layer_aws_helper --description “AWS Helper functions. Dependencies: Boto3 v1.12.14” --license-info “MIT” --zip-file fileb://lambda_layer_python_aws_helper.zip --compatible-runtimes python3.8
Windows:
Again, it’s recommended that you use a zip tool (such as 7-Zip) to zip up Lambda Deployment artifacts on Windows. Make sure that the ‘python’ directory is at the root of the archive.
Associating the Lambda Layer with your Lambda Function
Associate the aws_helper Lambda layer with your Lambda function using the same methodology as was shown in Example 1.
Importing the aws_helper library in your Lambda function
Within my Lambda function, the aws_helper library is imported using the layer’s Python filename, as shown on Line 4:
Once imported, Python functions defined within the library can be called as per usual. On line 12 we make a call to our ‘create_spoke_account_session’ function, using STS to assume a role in an external AWS account. With Lambda Layers providing a centralized location for custom libraries, any Lambda function with correct IAM permissions can utilize the aws_helper layer to create a session to any number of external accounts.
Lambda Layers — Things To Know
Having presented a couple of examples of how Lambda Layers can remove the need for duplicate code across Lambda functions, there are a few important restrictions that you should be aware of:
- A single Lambda Function can only reference up to 5 Lambda Layers. Much like architectural design, considerable thought should be given to the makeup and usage of Layers within a larger software application
- Lambda Layers do not remove the requirement for the total unzipped deployment package (Layers + Function code) to be 250MB or less
- Lambda Layers have an ordering structure, known as a merge order. The merge order of a layer for a function is defined in the same location where layers are associated. If the Lambda runtime discovers layers containing the same directories/files, directories will be merged and the files in the layer with the latest merge order will be applied
Summary
In software development, the ‘DRY’ principle aims to eliminate duplication at all costs. This can be particularly important for application source code, as maintaining instances of duplicate code can often be a logistical challenge. By introducing the concept of Lambda Layers, AWS made it possible to reduce duplication by sharing code between Lambda functions. As shown in the examples, effective use of Lambda Layers makes it possible to centrally manage dependencies and create libraries to be utilized by multiple Lambda functions. By considering the big picture and carefully planning how Lambda Layers can be used within your system, you should have no problem staying DRY when developing serverless applications in AWS!
