Creating Lambda Layers Made Easy with Docker: A Developer’s Guide
Streamline AWS Lambda development with Docker. Simplify dependencies, ensure consistency, and enhance serverless workflow.
Serverless architecture is popular for its scalability and cost-efficiency. AWS Lambda, a leading serverless platform, allows developers to run code without server management. Lambda functions serve as building blocks, while Lambda Layers efficiently manage shared code, libraries, and dependencies across multiple functions.
A Lambda layer is a distribution mechanism in AWS Lambda that allows you to manage and share common code, libraries, and dependencies across multiple Lambda functions. By creating a reusable layer, you can reduce duplication, simplify code maintenance, and promote efficient resource utilization in serverless architectures.
Creating Lambda layers can be tedious, especially when dealing with complex dependencies and ensuring compatibility. However, leveraging Docker simplifies the process to ensure compatibility and consistency across different environments. In this article, discover how to streamline your serverless development workflow by creating Lambda Layers using Docker.
Why Lambda Layers?
Scenario 1:
Imagine you have a Python Lambda function that requires a specific library, such as NumPy, for complex mathematical calculations. Instead of bundling the entire library with each deployment package, you can create a Lambda layer that includes the NumPy library. This lets you keep your deployment package lightweight while easily providing the necessary library to multiple Lambda functions.
Scenario 2:
Consider a scenario where multiple Lambda functions need access to a common set of utility functions. Rather than duplicating these utility functions across each deployment package, you can create a reusable Lambda layer containing the shared code. By centralizing updates and improvements to utility functions, maintenance is simplified, and duplication across functions is reduced.
Prerequisites
Install Docker and configure it correctly on your machine before following along in this article. This allows us to create an isolated environment for building the Lambda Layers and avoid potential dependency conflicts.
Setting up the Project
Start by organizing your project structure. Create a directory containing the necessary files, including the Dockerfile and requirements.txt. The Dockerfile serves as a blueprint for building the Docker image, while requirements.txt lists the Python dependencies required for the Lambda Layer. Our code structure will look like this:
Defining the Dependencies
Open the requirements.txt file and specify the Python packages your Lambda Layer requires. This ensures that the necessary dependencies are installed within the Docker container, enabling you to create a self-contained and reusable Lambda Layer. Here, we will use the requests library for demo purposes.
Building the Docker Image
The Docker image is the foundation for creating the Lambda Layer. Craft a Dockerfile that starts with a suitable base image, such as the official Python image. Install any system-level dependencies your Python packages require using the package manager (e.g., apt-get for Debian-based images). Then, copy the requirements.txt file into the Docker image and execute pip install to install the Python dependencies. So, your Dockerfile will look like this:
# Use the official Python runtime as the base image
FROM python:3.8-slim-buster
# Set the working directory in the container
WORKDIR /app
# Install any necessary dependencies
RUN apt-get update && \
apt-get install -y zip && \
rm -rf /var/lib/apt/lists/*
# Copy the requirements file to the working directory
COPY requirements.txt .
# Install the Python packages listed in requirements.txt
RUN pip install -r requirements.txt -t /opt/python/
# Set the CMD to zip the installed packages into a layer
# change the `requests-layer` to the LAYER_NAME variable as per create_layer.sh file
CMD cd /opt && zip -r9 /app/requests-layer.zip .When creating the Lambda layer, the zip command should be run from the
/optdirectory instead of the/opt/pythondirectory since the Lambda function will look for libraries inside thepythondirectory.
Creating the Lambda Layer
Here, I have created a shell script create_layer.sh that will run the Docker container using the image and bind mount the current directory. This allows the container to access the requirements.txt file and create the Lambda Layer accordingly. Within the container, execute the necessary commands to generate the requests-layer.zipfile containing all the installed dependencies.
Here is the code for the create_layer.sh file.
#!/bin/bash
# Set the directory where the Dockerfile and requirements.txt are located
DIRECTORY="$(pwd)"
# Change it as per your requirement
LAYER_NAME="requests-layer"
# Build the Docker image
docker build -t lambda-layer "$DIRECTORY"
# Run the Docker container to create the layer
docker run --name lambda-layer-container -v "$DIRECTORY:/app" lambda-layer
# create layers directory, if not created.
mkdir -p layers
# Move the zip file in layers directory.
mv "$DIRECTORY/$LAYER_NAME.zip" "$DIRECTORY/layers/$LAYER_NAME.zip"
# Stop the conainer
docker stop lambda-layer-container
# Remove the running conainer
docker rm lambda-layer-container
# Cleanup: remove the Docker image
docker rmi --force lambda-layer- Make the script executable by running the command:
chmod +x create_layer.sh. - Run command
./create_layer.shThis will createrequests-layer.zipand put insidelayersa directory. Don’t forget to change the variableLAYER_NAMEas per your requirements
Verifying the Layer
After the container has completed its execution, verify the generated requests-layer.zip file. Ensure that it includes all the required Python packages and dependencies, validating the successful creation of the Lambda Layer. So, your final folder structure will look like this:
├── create_layer.sh
├── Dockerfile
├── layers
│ └── requests-layer.zip
├── README.md
└── requirements.txtUploading the Layer to AWS Lambda
Open the AWS Management Console, and go to the lambda functions. Click Layers from the left side panel, then click Create Layer, as shown in the screenshot.
As per the screenshot below, fill in the details and upload the requests-layer.zip created by us, select the runtime Python 3.10 since we created a base image from Python 3.10, and click Create.
Once the layer is created, create a test lambda function by selecting runtime as Python 3.10 and paste the below code that will call the public API using requests library.
import json
import requests
def lambda_handler(event, context):
res = requests.get("https://restcountries.com/v3.1/all")
return {
'statusCode': res.status_code,
'body': res.json()
}Go to the bottom of the Lambda function details page, click on add layer, and fill in the details as per the screenshot, and the layer will be attached to your lambda function.
Now it’s time to test all the things we did, click on the Test button and add test event and click on test. You should get the below output.
Congratulations on successfully creating and incorporating a layer into your Lambda function.
This approach effectively minimizes redundancy and enhances the ease of code upkeep. Furthermore, you can leverage the capability to include multiple libraries within the requirements.txt file, enabling the shared lambda layers to be conveniently reused across various functions.
Key Learnings
By leveraging Docker, you can streamline the process of creating AWS Lambda Layers, making serverless development more efficient and reliable.
With Docker’s flexibility and AWS Lambda’s serverless architecture, you have the power to focus on writing business logic rather than worrying about infrastructure management.
Additional Resources:
You can get the code from this GitHub Repo.
Dive into the treasure trove of knowledge with Simform Engineering blog and unlock a world of captivating and insightful content.
