Snowflake
Published in

Snowflake

Getting your AWS Lambda Functions to work with Snowflake

A recent addition to the Snowflake feature-set was External Functions which greatly improves the extensibility of what you can do from a data engineering and data science perspective with Snowflake (more examples on these to follow…). Alternatively, you may have a scenario where you want to talk to Snowflake from an AWS Lambda using the Snowflake Python Connector.

When creating AWS Lambda Functions, you may need to import an external module beyond what is available by default — an example might be the requests module or the Snowflake Python Connector module mentioned above.

One approach to handling this is the Serverless Application Framework which is a tool for deploying your Lambda Functions to AWS, and is useful since it takes care of all the plumbing needed to get your function going. There are some great articles that explain the benefits of using this framework such as this one. There may be times however when you want to manually create your Lambda Function and therefore need to package your dependencies manually— this article focuses on this pattern.

Note — This requirement is not exclusive to Lambda Functions used with Snowflake, this applies to any Lambda that uses imports an external module.

We have 2 options to do this:

In this post, I will provide instructions on both; either work fine but the benefit of Lambda Layers are they are reusable by multiple Lambdas Functions.

Option 1 — Creating a combined package:

In this example we’ll be importing the requests module.

$ mkdir my-lambda-function
$ cd my-lambda-function
$ python3 -m venv venv
$ source venv/bin/activate

At this point you are now in your new virtual environment. We can now install the required dependency:

$ pip3 install requests

Once this is installed, add your lambda function file (in this example I’ll use snowflake_lambda.py as my filename). The next step is to bundle the dependencies and your python handler file together:

$ cd venv/lib/python3.8/site-packages
$ zip -r9 ${OLDPWD}/function.zip .
$ cd $OLDPWD
$ zip -g function.zip snowflake_lambda.py

The output will be the zip file needed to upload to the Lambda within the AWS CLI or Console.

Option 2— Using Lambda layers:

By using Lambda Layers we can upload modules that can then be imported into the Lambda function, the benefits to this are:

Note — using Lambda Layers does not overcome the overall size limitation of a Lambda imposed by AWS.

In a similar way to Option 1, we need to create a package to upload to AWS, but this time we are just uploading the module only, without the Lambda function itself.

To get started, we need to create our zipped package. In this example, I’ll use the requests module but the same would apply to any module (i.e. snowflake-python-connector).

$ mkdir -p temp/python
$ cd temp/python
$ pip3 install requests -t .
$ cd ..
$ zip -r9 ../requests.zip .

In the directory above your temp folder you will now have a requests.zip file that contains all files part of the requests module.

To upload to AWS, using the console do the following:

Alternatively you could use the AWS CLI:

$ aws lambda publish-layer-version --layer-name requests \      
--zip-file fileb://requests.zip \
--compatible-runtimes python3.8

Using Docker to build your package

If you are building your package on MacOS, then you may hit an error that manifests as an invalid ELF header message. This occurs because the Lambda Function runs on Amazon Linux, but you are compiling the package in a non-linux environment (i.e. MacOS).

In order to overcome this you need to build your package within a Linux environment. The quickest way to do this is using Docker to spin up a container locally and perform the same steps as above in (1) or (2). Once the container is spun up, we will need to install a few tools, but will be able to perform the same commands.

To do this, in the terminal perform the following:

$ docker run -v <directory with snowflake_lambda.py>:/lambda -it --rm ubuntu

Notes:

Once this has executed (Docker may download the ubuntu image if needed, so may not be instant), you will be in a terminal session within the container:

root@7b6ea21bd7a5:/#

Next we need to install some packages:

$ apt-get update
$ apt-get install python3-pip
$ apt-get install zip
$ apt-get install python3-venv

Note — you don’t need to do line 4 if you are doing the Lambda Layer approach.

You will now find yourself in your working folder. From here you should be able to then execute Option (1) or (2) and you will have a zip file, compiled for Linux to be uploaded to AWS.

Example — Bundling a single deployment package with the Snowflake Connector

$ docker run -v <your dev directory>:/lambda -it --rm ubuntu$ apt-get update
$ apt-get install python3-pip
$ apt-get install zip
$ apt-get install python3-venv
$ cd lambda$ mkdir my-lambda-function
$ cd my-lambda-function
$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install snowflake-connector-python
$ cd venv/lib/python3.8/site-packages
$ zip -r9 ${OLDPWD}/function.zip .
$ cd $OLDPWD
$ zip -g function.zip snowflake_lambda.py

Example — Building a Lambda Layer with the Snowflake Connector

$ docker run -v <your dev directory>:/lambda -it --rm ubuntu$ apt-get update
$ apt-get install python3-pip
$ apt-get install zip
$ cd lambda
$ mkdir -p temp/python
$ cd temp/python
$ pip3 install snowflake-connector-python -t .
$ cd ..
$ zip -r9 ../snowflake-connector-python.zip .

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Simon Coombes

Director of Data Science Services @ Snowflake ❄️. ML Enthusiast, IoT Builder, and Maker/Inventor when I have time.