Lessons learned in serverless

Welcome to the first post in the series discussing some small lessons learned whilst developing on AWS. For the context of the series and what we’re looking to achieve, please read the introductory post An introduction to some small lessons learned developing on AWS.

This post is going to focus on serverless. First, we’re going to define what serverless is, both the approach and also the framework. We’re then going to look at some of the lessons we’ve learned whilst developing serverless applications on AWS, using the Serverless framework.

What is serverless?

To continue the post, let us define what serverless architectures are. Martin Fowler's bliki begins with this definition:

Serverless architectures refer to applications that significantly depend on third-party services (knows as Backend as a Service or “BaaS”) or on custom code that’s run in ephemeral containers (Function as a Service or “FaaS”)

For the purposes of this post, we’re going to focus on serverless as Functions as a Service. The approach of using Functions as a Service solves a number of problems, namely high availability, automatic scaling and no server management. It is pay per use, meaning you only pay for the compute time of the function when it is invoked. This can significantly reduce costs when comparing to running either full-blown instances or even long running containers. It forces a design approach to solve one problem at a time, and to borrow a term from Domain Driven Design, enforces a strictly bounded context at the function boundary. Also, you could consider whether taking a serverless approach to your architecture increases feature delivery velocity.

Also, there is the Serverless framework which is a toolkit for deploying and operating serverless architectures. This allows us to define our infrastructure (serverless architecture) in code. The Serverless framework is provider agnostic and allows for rapid deployment. An interesting side note here is serverless by the numbers 2018 data report which highlights the languages used in AWS. Of particular interest for us was the usage of dotnet core which was less than 2%. Though the numbers are low, there is some encouragement in the post Comparing AWS Lambda performance of Node.js, Python, Java, C# and Go showing dotnet core running at significantly lower runtimes.

There were two specific use cases we highlighted as applicable uses of the serverless approach. These are hosting static websites in S3, and running distinct units of code as functions in Lambda invoked by CloudWatch Events and API Gateway. For the remainder of this post, we will be focusing on Lambda.

Lessons learned in AWS

Lambda is AWS’ offering of Functions as a Service. It can accept invocation from a number of other AWS offerings such as API Gateway, CloudWatch events, Kinesis streams, other Lambdas and many more. We were particularly interested in running our Lambda functions via API Gateway and CloudWatch events.

The first thing we found to be extremely important is logging. Creating Lambda functions comes with logging built in and it’s really important to utilise this immediately.

Given the nature of Lambda and the fact that there isn’t a server to log in to. Logging becomes a key resource in your ability to debug the functions that have been deployed. When you log within your Lambda function, it logs directly to CloudWatch.

Logs in CloudWatch

Due to the nature of the solution, we also had to attach our Lambda functions to a VPC in order to gain access to some internal resources that we didn’t want to expose to the internet. This had some side effects:

• We lost internet access within the VPC
• We were severely hit by cold starts

We found out we needed internet access immediately when we attached the Lambda function to the VPC. This manifested as we were querying SSM for secrets used within the application. By default, Lambda functions that are created unattached have internet access, but as soon as they are added to a VPC you need to ensure that a NAT gateway is provisioned to give the Lambda function internet access.

In our case, there was a second option, which is to use VPC Endpoints. A secure endpoint that allows you to securely connect your VPC to another AWS service.

VPC Endpoint selection interface

Secondly, and perhaps our biggest issue was the effect attaching our Lambda function to the VPC had on the functions cold start. There has already been a lot written about Lambda cold starts and we’ve certainly not solved the problem itself. Understanding what a cold start is, and why it is significantly affected by attaching it to a VPC allowed us to look at strategies to mitigate this. The lesson here is that it is so much longer because when the function is attached to a VPC, it has to wait to be allocated an ENI with the assigned VPC subnet.

In order to keep our functions warm, we invoked each of the functions periodically from a different lambda function. This request contained a header which signified it was specifically to keep the function warm.

Due to the billing profile of Lambda, this added a very small cost overhead (less than $0.10) and despite being a little ugly, this gave us the desired impact. We still had issues with deployments and managing the impact of deployments, but will discuss this a different time.

Lessons learned in Serverless

The Serverless framework is a toolkit for deploying and operating serverless architectures. It supports some of the largest cloud vendors, including AWS, which is the location of our serverless deployments.

Cloud vendors supported by Serverless

You can create a new serverless application easily by running the command

sls create --template aws-csharp --path ServiceName

This command currently creates a dotnet core 1.1 application and all of our functions are written in dotnet core 2.0. You can easily remedy this by either updating the runtime in your serverless.yml file to dotnetcore2.0 or by creating the project from scratch and copying a serverless.yml file across.

A good early lesson is to understand variables, which allows you to tailor the deployment to the environment it is targeting. There are two methods of inserting variables into you serverless.yml script, via file or by environment. ${file(./serverless.env.yml):${opt:stage}.SECURITY_GROUP} or ${env:${self:provider.stage}variable}.

Embedded within these two approaches, we can also see a reference to stage. This is set when invoking the serverless deploy command serverless deploy -v -stage production. You can declare a default to use for the stage in the serverless config as well stage: ${opt:stage, 'dev'} where dev will be the default if the stage is not included in the serverless deploy command.

Our original reason for using the Serverless framework was to use the ability to enable cors on the API Gateway. This was a simple case of including cors: true in the declaration of the events that may invoke the lambda function:

events:
  - http:
      path: hello
      method: get
      cors: true

This is enough to create an API Gateway endpoint, with cors enabled which in turn invokes the function you have created.

Another event type we utilised was via CloudWatch Events. Again, this was a simple case of declaring this as an event that can invoke the lambda function:

events:
  - schedule:
      rate: cron(0/5 7-19 ? * MON-FRI *)
      name: a-cloud-watch-event
      description: 'Invoked every five minutes, 7 while 7, Mon - Fri

At a global level, we can set IAM permissions for services the function may need to access. We can also specify the VPC and subnets to attach the Lambda to should we need to access other internal services within the VPC. This should be approached with caution and only done when necessary. As described earlier, attaching the Lambda to a VPC significantly increases its cold start duration.

All of this can be seen in the sample serverless script:

What we’ve touched on barely scratches the surface of what can be achieved using the Serverless framework. It can be viewed as an Infrastructure as Code approach to developing your serverless architectures and manages the resultant CloudFormation configurations that are generated. The Serverless framework stores these in an S3 bucket and initiates the deployment upon the sls deploy command.

One of our current challenges is assessing how many functions to include in one serverless.yml config file. Our current assumption is that functions in a similar use case should be grouped, but the introduction of VPC attachment is challenging this. When we deploy these groups, each function is subsequently exposed to the cold start again which may not be acceptable. There are two ways we believe this could be handled. Each function could be split into its own serverless.yml config. This would also have the advantage of applying IAM roles only where required, in line with least privilege. The alternative is to engineer a blue-green deployment, whereby application traffic is only routed to warmed up functions.

In conclusion

Hopefully, this has helped to highlight some of the challenges we have faced on our journey to understanding serverless architectures. We’ve looked at the difference between what a serverless architecture is, and what the Serverless framework offers. We looked at some of the lessons the team has learned developing functions for AWS. Finally, we looked at some of the small lessons we learned in developing our serverless applications using the Serverless framework.

Next up, we’ll be looking at containers, and some of the lessons we’ve learned using AWS ECR, ECS and Docker.