File Storage on AWS — Designing Lift

This article is part of a series on Lift, an open-source project that simplifies deploying serverless applications.

Thanks to tooling like the Serverless Framework, deploying serverless apps to AWS is easier than ever. However, some parts are still harder than they should be.

This is why we started Lift, an open-source project that simplifies deploying serverless apps. Lift will be distributed as a plugin that adds new features to the Serverless Framework. The project is currently work in progress, with a first beta planned for May 2021.

Last week, we introduced Lift’s Static Website feature 🎉 ! Let’s continue this series with the File Storage component. In this article, you’ll learn:

• How to deploy production-ready storage solutions on AWS today
• How we plan to simplify that with Lift.

Searching for a storage solution on AWS, you’ll first stumble upon a choice of 10 different services! In addition, those services have an always-increasing amount of available configuration, with AWS S3 documentation going well over the 1,000 pages. Not so simple for the Simple Storage Service.

With the number of services and range of configuration options available it’s not fair to expect all developers to be an expert in all of them. We need to ensure security, reliability and cost efficiency through better abstractions for those storage solutions.

Serverless File Storage naïve approach

Among the 10 storage solutions, only two are dedicated to serverless workloads. The main difference between S3, the Simple Storage Service mentioned beforehand, and EFS, the Amazon Elastic File System, is the protocol to use: NFS or HTTP.

In most cases, S3 is the perfect go-to solution and most use cases can be developed using S3. With the help of the SDK, you can easily interact with this storage solution from your Lambdas.

S3 having an HTTP API, it can be interacted with from your browser as well as from non-compute AWS services like CloudFront for Content Delivery Network, Glue for Extract Transform Load or QuickSight for Business Intelligence.

Finally, new files, updates and deletion happening in your bucket are streamable in the form of events, that you can use to trigger a dedicated Lambda, dispatch a message to an SNS topic or even queue a message in SQS.

Map of available S3 integrations

Adding a new S3 bucket in your infrastructure requires only 2 simple lines of CloudFormation YAML, all properties being optionals.

MyBucket:
  Type: AWS::S3::Bucket

However, this default configuration is not ideal.

What needs to be considered when using S3 ?

• Security: Ensuring the bucket is private (not accessible on the internet!) by default, and its content is stored encrypted
• Cost optimized: Ensuring the content is stored with the lowest cost possible for the desired availability and response time, leveraging the various S3 billing options, and providing an autonomous mechanism to move files from one pricing to another depending on the frequency at which it’s accessed.
• Reliable: Preventing accidental overwriting or deletion using S3 versioning. To avoid extra costs caused by versioning, S3 should be configured to automatically prune older versions.

To automate this with CloudFormation, we would need over 40 lines of YAML… All that for a simple bucket!

This should be simpler!

Deploying storage solution with Lift

Lift encapsulates and abstracts away this minimal configuration by deploying buckets with encryption, versioning, cost-optimized intelligent tiering, and required metrics and analytics to monitor.

Lift will also automatically setup permissions that allow the Lambdas of your serverless application to interact with this bucket.

service: myapp
provider:
  name: aws
functions:
  # AWS Lambda functions
  # [...]
# new storage root property to define your different stores
storage:
  # you can name your storage however you'd like to
  invoices:
  avatars:
    archive: false # Deactivate archive default feature

Lift will generate and deploy the required CloudFormation resources when you run the serverless deploy command.

Lift will provide you with detailed information and recommendations to update your bucket configuration on any subsequent deploy command based on your actual usage of your storage component.

Configuring storage solution is a small part of what we are working on with Lift.

We’re looking for feedback on the File Storage feature: get involved in the GitHub discussion.

If you want to support Lift development, star or watch Lift on GitHub.