Fun with AWS Lambda Layers and Serverless

Barend Bootha
Jun 22 · 4 min read

This article assumes you know what AWS Lambda layers are and that you’re familiar with Serverless, if not, have a read here.

How we screwed our selfs

There is a saying under devs “never deploy on a Friday”, ignoring this because I know what I’m doing is how this story starts 😂 or at least I thought I knew what I was doing.

The title to this paragraph is total click-bait, I’m (currently) in a lucky position that the product I’m working on, is not yet in production and is being built. But if it was… We would have been screwed, tears would have rained, Friday beer-o-clock would have been missed and the week ended on a low.

It’s Friday afternoon and I wanted to roll out a change in a Lambda, all the pipelines was green, it WOMMED (Works On My Machine) just fine, all roads lead to Rome, merge and walk away (Again this cockyness is how you get burned).

As my Lamba was deploying via Serverless, CloudFormation spat the dummy, there is a valid error and the stack started rolling back. But then

Yikes…

Layer versions and Serverless

It turns out, Serverless by default do not retain versions of your Lambda layers, this setting is set to false. See the Serverless Layer documentation

# serverless.ymlservice: myService
provider:
name: aws

layers:
hello:
...
retain: false # optional, false by default. If true, layer versions are not deleted as new ones are created

If you’re running a Continues Deployment (CD) pipeline, one thing you always always need, is being able to rollback to a previous version.

It turned out as CloudFormation was rolling back, my Lambda required version X of the Lambda Layer referenced, that layer version was no longer available, it got deleted when we started rolling out the new version of the Lambda.

I then had to deal with the fallout of the CloudFormation rollback and it’s just not fun, or easy to fix.

Moving forward and deploying often

The reason retain is false by default is probably because there is no good story to maintain these versions and if you’re deploying often, and you should, there might be 100’s of versions of your layer floating around.

What Serverless don’t provide is good practice around maintaining the versioning of your layer, nor does it check if what your deploying is different to what’s already up there (It does for your lambda 👍👍👍👍).

Deployments should be a hands off approach, if your devs have to intervene or manually path out deployments (choosing what needs to be deployed with what) you need to rethink your deployment process. However I think it’s totally acceptable to gate deployments (Having only certain people with elevated permissions deploy to production or the like). So moving forward lets automate the versioning and only deploy when it’s needed.

Hash the layer

My fix around this is to set a hash on the deployed lambda layer and compare it on the next deployment.

In one of my pipeline steps we build the lambda layer with a sls package command.

As the artefacts from the build process is passed over the the deployment pipeline. I check the Hash of the package, against that of the last deployed layer (PS: This would only work for new deployments, a rollback would receive a desired version etc…).

Simple hash check of deployed layer vs what was built

When the layer is deployed I store the Hash in the description, like so.

Storing the hash in the description

Many ways to skin a cat

Or so they say, I love cats too much… 😹

My solution is just one way, not necessarily “the way”, another option would be to store the uploaded Layer version in thepackage.json. If the package with the same version is already up in AWS ⛈, then you’re okay and don’t need to deploy again.

At the end of the day, you need to maintain the version of what was uploaded against that of what you’ve got in code.

In our case, the hash method made sense as we’re using a mono repo with a hoisted node_modules, and then dynamically construct the layer from the production dependencies.

Best practices

  1. Lambda’s and Layers should not live within the same stack. And when I say stack, I’m referring to the serverless.yml file.
    It’s best to keep these seperate as they won’t always have the same deployment cadence. Your lambda’s code will change more often than the layer.
  2. Explicitly set the version of the lambda layer being used.
    Once you’ve uploaded your layer, get the version of that package and explicitly pass it into the Lambda that’s using that layer.

In summary

Lambda layers have many benefits, I don’t need to outline that here. However you do need to take care of how you use them. And by all means retain the versions of your deployed layers otherwise you might just get burned like I did 🔥.

Barend Bootha

Written by

Backend engineer and DevOps practitioner, current focus is a NodeJS stack hosted in AWS

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade