Microservices with Serverless & AWS Lambda Layers — A Walkthrough
Since Microservices are becoming more popular, the technology used to create these architectures is becoming more important as well. We want to keep these architectures both scalable and elastic. So, making use of a cloud service like AWS Lambda is gaining popularity as well!
For efficient development & deployments, our team at Zoi uses Node.JS 10 and the Serverless framework for our projects.
I would describe a Microservice as a small service that works in a set and shares business logic. The architecture must keep the system very flexible, so that, for instance, a single service can be updated while the rest of the system keeps running. That means the service runs self-contained and blue-green deployments are more easier to handle.
Dependencies should also be shared in the services, which simplifies the development of these systems. This article describes how we can make use of the powerful Serverless Tool combined with AWS Lambda Layers and API Gateway to create mid-sized to big enterprise-grade microservices-based architectures.
Reason / Up- and Downsides.
Is there a reason, that requires switching to an architecture with Layers in Lambda?
Yes, when your systems grow too big for a single deployment. The CloudFormation template, which will be created by serverless is limited to 200 resources. Each Lambda function with an HTTP endpoint, for example, creates 4 resources in your deployment.
Splitting these services will allow us to be more flexible and also make the structure and error more complex to handle. The main benefit of using a Layer in Lambda is to share dependencies and business-logic between different functions.
The limits of Layers are defined in the AWS Documentation (Cited 28.11.2019):
- A Lambda can hook up to only 5 Layers at once
- Like Lambda, it’s not allowed to be bigger than 250MB
- Deployed per Account
Enough with the theory, let’s show with an example how AWS Lambda Layers works in practice. The steps can also be looked up in the Repository (Branch step_1, step_2 etc.). All the serverless.yml files will be created manually.
First, we will create a simple Lambda and a redirect on the API Gateway through a domain. This is the example for a Serverless Hello World Application (Example Branch):
After that, we create the Layer that will contain our dependencies. Simply create the folders, and run “npm init” in each folder (Example Branch):
Here, we can define all the node packages as usual. In this very example, it’s only the popular node.js library “moment” that is used for formatting dates and our business dependencies. It’s possible to install your own dependencies with npm by running “npm install ../business-dependencies” and moment with “npm install moment” in the directory.
We will also create a serverless.yml file in the root of the project that looks like this:
Here, we define the layer, which needs a name, the path of the layer, and the compatibleRuntimes. You can imagine that the layer works like a docker image. When a layer is hooked onto a Lambda, it will now load all the dependencies for it.
Now we need to define the layer in the lambda, we will use a variable (that could be generated by e.g. Terraform) and attach it for multi stage deployments in the Lambda. In this example, we load a json file and use the account_id property (Example Branch):
Since the term “latest” is listed as the version, we will need a plugin called serverless-latest-layer-version, which will enable the program to always use the latest layer version of the dependency layer. That’s it! Now, we can deploy the layer and then the latest version of the layer afterwards. We will run `serverless deploy — stage dev` in the directory of the dependency layer and then again in your lambda to update it with the current version of the Lambda. The full example can be looked up at GitHub.
Thanks for reading this! I hope you can now see how flexible, extensive and awesome this architecture can be! For example, updating a single api currently takes about 2 minutes, even with only uploading around 3kB of Code. Since you can hook up to 5 Layers on a single Lambda, we are able to split our dependencies in smaller chunks. So, we use only relevant libraries for our specific Lambdas. The complete API takes about 10 Minutes to perform a full update, which will likely be improved as it gets developed further.