Are AWS Lambda Layers dangerous?
I’ve been looking into AWS Lambda and the AWS Lambda Layers feature got my attention.
Overall, from the architecture perspective, this looks like a nice to have optimization feature, like Docker Image Layers. There are a lot of articles explaining how great the layers are and what a nice way to handle dependencies it is.
However, after digging deeper, I see very limited benefits of using layers and actually, there are more serious and even dangerous drawbacks.
The feature is advertised as a good way to manage dependencies, but in my opinion, it makes dependencies management much harder and error-prone. Each technology has its own way of managing dependencies. I’m working with JavaScript and Node.js, so I will be talking about npm.
A Dependency Hell
In npm you have a manifest file (package.json) which specifies all the dependencies your project use and more importantly the version ranges for those dependencies. But, if you are using Lambda Layers to extract some dependencies into its own layer you can’t use npm anymore to manage them. You are effectively breaking the dependency management system, which is complex enough on its own accord, into two sub-systems, that require manual coordination. Now, your project is not listing some dependencies explicitly and rather depend on their implicit presence in the global context (external layer). This introduces an implicit (hidden) dependencies, that could be easily broken.
Now consider, that you have multiple Lambda Functions which need to use various versions of the same dependency. For example, the new major version of some dependency was released. Now, you can’t update the dependency in only one of your functions, you will need to update all the functions at once. And if you would forget to update one of them, but would publish a new layer, it could break them in runtime or affect their behavior in an unpredictable way. Of course, you could use various versions of the various layers, but you can’t have more than 5 layers (AWS limitation) and this will not give you enough granularity.
A Broken Encapsulation
Also, by separating your package into multiple isolated layers, you are effectively breaking an encapsulation. This means that your package can’t be treated as a single self-sufficient unit of code anymore.
You will not be able to clone your package from the VCS individually or extract it from the published ZIP-archive, install the dependencies using npm install and run it locally because the extracted dependencies would be missing. You will need more complex tooling to work with the packages that use Lambda Layers. Of course, you can leave those dependencies in your project as is, but then it will require more complex deployment mechanisms to prevent those dependencies from being added to the Lambda Function archive, which is additional complexity with manual configuration for each package. And this will not protect you from the versions mismatch.
Also, it is a great idea to use Webpack with the tree-shaking to build your package (Lambda Function) into a single bundle (a single file that includes your code as well as the third-party one) instead of deploying all your source files and the dependencies as a big tree of files. This would dramatically reduce the size of your Lambda Function archive because only the used libraries and symbols from those libraries will be added to the bundle. But, if you are using Lambda Layers, this optimization will not work, because your code will be isolated from the library code, and you wouldn’t know what code to remove from the library when you publish the layer. And this will become even more important in the future because more and more libraries will be published in ESM format, that supports tree-shaking, making your bundles even smaller.
No matter how I look at it, I see that Lambda Layers only introduce a dependency hell with hard to avoid regressions, break the Lambda Function encapsulation, require more complex tooling to work with the code and prevents the build optimization practices from being effectively implemented.
In conclusion, if you are working with JavaScript, Node.js and npm I would strongly advise to avoid usage of the AWS Lambda Layers and publish each Lambda Function as a single self-sufficient package with explicit dependencies and using bundling with dead code elimination techniques to publish your code as a single optimized file.
And if you are using some other technology, please consider the drawbacks, that I’ve mentioned in this article, probably this would apply to your technology as well.
However, in defense of the AWS Lambda Layers feature, I see a single case where this would be useful. If you are depending on some very big library (which is a rare case in npm) and you are absolutely certain that all your Lambda Functions will always use the same version of it, then extracting it into a separate layer would be a good thing to consider. But, I believe that this would be a very rare situation in Node.js development, considering the npm ecosystem.
