How to use AWS Lambda Layers
You have more than 20 different AWS Lambda functions that use the same set of pip/bundle installed and custom libraries. And someone just told you about the new AWS Lambda ‘Layers’ and now you can’t wait to use them! But after some basic experimentation you end up having a lot more questions than answers and you’re frantically googling up keywords but in vain. I was in the same boat and so I decided to jot down my findings here.
What is a Lambda Layer?
From the introductory post:
Now, you can put common components in a ZIP file and upload it as a Lambda Layer. Your function code doesn’t need to be changed and can reference the libraries in the layer as it would normally do.
You can think of it as an read-only filesystem that is mounted at ‘/opt’ and which can be used across your lambdas. The interesting bit is that you can mount multiple filesystems at the same path and that’s where it gets confusing. Lets start with a basic Ruby example to see how we can use AWS Lambda Layers. The Ruby runtime support was recently added to AWS. I’ve written hundreds of Python and Go lambdas but it is my first time using the Ruby runtime in AWS Lambda — so this would be fun!
1. Single Layer
On my mac, I have a folder ‘ruby_libs’ which contains a single file ‘custom_func.rb’
which contains this 3 line function.
And I zip it:
Lets create a Lambda Layer called ‘ruby1’
and a simple lambda which uses this layer. We add a line to print the contents of ‘/opt’ and another line to call the custom function we created
and add to our custom layer:
It is time to run the lambda! If we invoke the lambda (with any input event) the logs show us the contents of the ‘/opt’ directory and confirm that our ‘custom_func’ was available to the Lambda
2. Multiple Layers
Lets create another library function ‘custom_func2’ by creating a new file
with a simple print statement
and then zipping it:
Create the layer and add to our lambda — you know the drill!
In our lambda, if we require the new function, and call the 2 functions .. we’ll be able to!
3. Avoiding absolute imports
Who wants to import every single file each time they want to use some shared code? Not me! That’s why we have ‘PATH’ environment variables for different languages: GOPATH for golang, GEM_PATH for ruby, etc. If you read the Layers documentation, you would have stumbled upon this statement:
You can move runtime dependencies out of your function code by placing them in a layer. Lambda runtimes include paths in the
/optdirectory to ensure that your function code has access to libraries that are included in layers.
And for Ruby, we this mentioned:
So the directory ‘/opt/ruby/lib’ is already set at ‘RUBY_LIB’ in the lambda and we can leverage this to put shared functionality and not use absolute imports. Lets do this
Put the shared code in ‘ruby/lib/’ and zip it.
Add the 3rd layer
Remove the absolute imports from the top of the file
and invoke the lambda:
4. Using module installers like bundle or pip
With Ruby 2.5.0 if you bundle-install gems in a Gemfile with the command
bundle install --path=. the directory structure created would be
Lets use a simple Gemfile and bundle install it with
bundle install --path=.
Once installed, all the gems can be zipped together:
Now, for Lambda Layers the second path that can be set for automatic loading of libraries in the ‘GEM_PATH’. From the documentation:
Important: AWS Lambda layer swaps the ‘gems’ and ‘2.5.0’ part in the expected path!
So, lets override this path by setting the environment variable to the standard one used by Ruby:
If we update the lambda to print if we were successfully able to require the 2 gems
and run it
we find that, yes, we were able to import these gems!