Exploring the Power of AWS EFS with Lambda:

Unleashing Scalable and Serverless File Storage

Photo by Mr Cup / Fabien Barral on Unsplash

One of the such powerful combinations in Amazon Web Services (AWS) is Elastic File System (EFS) integrated with AWS Lambda, the serverless computing service.

AWS EFS provides highly available and scalable file storage, while Lambda allows executing code without the need to manage servers. Together, they enable developers to efficiently handle file-based workloads in a serverless architecture. Such as:

• Sharing large software packages or binaries that exceeds the allowed size for the Lambda layers.
• Processing large video files
• Archiving / extracting large number of files

EFS is designed to effortlessly scale on-demand, accommodating petabytes of data while automatically adjusting its size as files are written or deleted. When paired with Lambda, this dynamic integration ensures your code has low-latency access to a persistent file system, even after the function has completed its execution.

Lambda automatically mounts EFS file systems during the preparation of the execution environment, resulting in minimal latency during the function’s initial invocation. Since the EFS mount is already available and ready to access, subsequent invocations benefit from an already warm execution environment.

In this blog post, we will dive into the seamless integration of AWS EFS with Lambda and explore how can we create flexible, efficient, and cost-effective solution for file processing and storage in the cloud. We will create a Lambda function that will sync an EFS file system with an S3 bucket with whenever a new object is uploaded to / removed from the bucket.

Creating and Configuring the Elastic File System

Navigate to the EFS console and choose Create File system. Specify the name and your preferred VPC for the new file system and click Create.

Then create a security group that will allow access for itself on 2049 port. In this article we will use the following security group:

We will use this security group also in our Lambda function.

After that, get back to the EFS console and select the newly created file system details. On the Network tab, click Manage and choose the related security group:

And then, get back to the file system details. On the Access points tab choose Create access point and enter the following parameters and choose create:

This configuration creates a file system with open read/write permissions. You can checkout the related AWS documentation for further details.

Configuring AWS Lambda to use EFS

In order to work with lambda functions that access EFS we need to deploy our function within a VPC. You can checkout the related AWS documentation about configuring the lambda function in VPC.

Before ± enable it to access to the VPC and EFS.

VPC access: You can attach the AWSLambdaVPCAccessExecutionRole policy (which is AWS-managed) to your function.

EFS access: You can create an inline policy with the following content to access the specific file system.

{
    "Action": [
        "elasticfilesystem:DescribeMountTarget*",
        "elasticfilesystem:ModifyMountTargetSecurityGroups",
        "elasticfilesystem:CreateMountTarget",
        "elasticfilesystem:DeleteMountTarget",
        "elasticfilesystem:ClientMount",
        "elasticfilesystem:ClientWrite",
        "elasticfilesystem:CreateTags",
        "elasticfilesystem:DeleteTags"
    ],
    "Resource": "arn:aws:elasticfilesystem:<region>:<account-id>:file-system/<file-system-id>",
    "Effect": "Allow"
},
{
    "Action": [
        "elasticfilesystem:ClientMount",
        "elasticfilesystem:ClientWrite"
    ],
    "Resource": [
        "arn:aws:elasticfilesystem:<region>:<account-id>:access-point/<file-system-access-point-id>"
    ],
    "Effect": "Allow"
}

Note: Here we would use the managed AmazonElasticFileSystemClientFullAccess policy but it is a better practice to use more restrictive policy to control access to the specific file system resources.

Attach the file system to the function. You can specify the file system, access point and local mount path like the following (mount path should start with /mnt):

You can see the example code for this function that is written in Python:

import os
import logging
import traceback
import sys

import boto3
from enum import Enum

logger = logging.getLogger()
logger.setLevel(logging.INFO)

region = os.getenv("AWS_REGION")
session = boto3.Session(region_name=region)
efs = session.client('efs')
boto_s3 = session.client('s3')

efs_mount_point = '/mnt/efs'
sys.path.append(efs_mount_point)


def download_file(bucket, s3_key):
    local_path = f'{efs_mount_point}/{s3_key}'
    logger.info(f'Downloading file {s3_key} to local as {local_path}')
    boto_s3.download_file(bucket, s3_key, local_path)


def remove_file(s3_key):
    local_path = f'{efs_mount_point}/{s3_key}'
    logger.info(f'Removing local file {local_path}')
    os.system(f'rm -f {local_path}')


class Event(Enum):
    OBJECT_CREATED = "ObjectCreated"
    OBJECT_REMOVED = "ObjectRemoved"


event_map = {
    Event.OBJECT_CREATED: download_file,
    Event.OBJECT_REMOVED: remove_file
}


def lambda_handler(event, context):
    try:
        logger.info(f'Handling event: {event}')
        record = event.get('Records')[0]
        event_name = record['eventName']
        s3_record = record['s3']
        object_name = s3_record['object']['key']
        bucket = s3_record['bucket']['name']
        if event_name:
            event_enum = Event(event_name.split(':')[0])
            event_function = event_map[event_enum]
            event_function(bucket, object_name)
    except Exception:
        message = f'An exception has occurred: {traceback.format_exc()}'
        logger.error(message)
        return {
            'statusCode': 500,
            'body': message
        }

It will be triggered on:

• A new object is created on the bucket
• An object is deleted from the bucket

and it will download or remove the related object from the file system accordingly. We can define the related triggers by defining the source S3 bucket, object prefix/suffix using the AWS console just like below:

Lambda triggers for S3 object put/delete events

We have integrated an EFS file system with our Lambda function successfully.

Thanks for reading. If you have questions or comments regarding this article, please feel free to leave a comment below.