This article assumes you know what AWS Lambda layers are and that you’re familiar with Serverless, if not, have a read here.
How we screwed our selfs
There is a saying under devs “never deploy on a Friday”, ignoring this because I know what I’m doing is how this story starts 😂 or at least I thought I knew what I was doing.
The title to this paragraph is total click-bait, I’m (currently) in a lucky position that the product I’m working on, is not yet in production and is being built. But if it was… We would have been screwed, tears would have rained, Friday beer-o-clock would have been missed and the week ended on a low.
It’s Friday afternoon and I wanted to roll out a change in a Lambda, all the pipelines was green, it WOMMED (Works On My Machine) just fine, all roads lead to Rome, merge and walk away (Again this cockyness is how you get burned).
As my Lamba was deploying via Serverless, CloudFormation spat the dummy, there is a valid error and the stack started rolling back. But then
Layer versions and Serverless
It turns out, Serverless by default do not retain versions of your Lambda layers, this setting is set to false. See the Serverless Layer documentation
# serverless.ymlservice: myService
retain: false # optional, false by default. If true, layer versions are not deleted as new ones are created
If you’re running a Continues Deployment (CD) pipeline, one thing you always always need, is being able to rollback to a previous version.
It turned out as CloudFormation was rolling back, my Lambda required version X of the Lambda Layer referenced, that layer version was no longer available, it got deleted when we started rolling out the new version of the Lambda.
I then had to deal with the fallout of the CloudFormation rollback and it’s just not fun, or easy to fix.
Moving forward and deploying often
retain is false by default is probably because there is no good story to maintain these versions and if you’re deploying often, and you should, there might be 100’s of versions of your layer floating around.
What Serverless don’t provide is good practice around maintaining the versioning of your layer, nor does it check if what your deploying is different to what’s already up there (It does for your lambda 👍👍👍👍).
Deployments should be a hands off approach, if your devs have to intervene or manually path out deployments (choosing what needs to be deployed with what) you need to rethink your deployment process. However I think it’s totally acceptable to gate deployments (Having only certain people with elevated permissions deploy to production or the like). So moving forward lets automate the versioning and only deploy when it’s needed.
Hash the layer
My fix around this is to set a hash on the deployed lambda layer and compare it on the next deployment.
In one of my pipeline steps we build the lambda layer with a
sls package command.
As the artefacts from the build process is passed over the the deployment pipeline. I check the Hash of the package, against that of the last deployed layer (PS: This would only work for new deployments, a rollback would receive a desired version etc…).
When the layer is deployed I store the Hash in the description, like so.
Many ways to skin a cat
Or so they say, I love cats too much… 😹
My solution is just one way, not necessarily “the way”, another option would be to store the uploaded Layer version in the
package.json. If the package with the same version is already up in AWS ⛈, then you’re okay and don’t need to deploy again.
At the end of the day, you need to maintain the version of what was uploaded against that of what you’ve got in code.
In our case, the hash method made sense as we’re using a mono repo with a hoisted node_modules, and then dynamically construct the layer from the production dependencies.
- Lambda’s and Layers should not live within the same stack. And when I say stack, I’m referring to the
It’s best to keep these seperate as they won’t always have the same deployment cadence. Your lambda’s code will change more often than the layer.
- Explicitly set the version of the lambda layer being used.
Once you’ve uploaded your layer, get the version of that package and explicitly pass it into the Lambda that’s using that layer.
Lambda layers have many benefits, I don’t need to outline that here. However you do need to take care of how you use them. And by all means retain the versions of your deployed layers otherwise you might just get burned like I did 🔥.