Running Haskell (or anything, really) on AWS Lambda

So you want to move your application onto AWS Lambda, because maintaining servers is a pain and you only want to pay for your code when you really need it to run. You’ve spent weeks trying to set up a self-healing architecture that automatically replaces failed EC2s and scales out for you and it just doesn’t work. Lambda looks promising, but the sticking point is language: why can’t I run any programming language I want?

Good news: you can, and there are no weird restrictions on your code that you need to work around!

TLDR: Compile your program on an EC2 instance based off of this machine image (amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2). Call the executable from a Python script. Copy in any dynamic link dependencies, and patch the executable to point to them. Zip up the whole thing and deploy it to AWS Lambda.

While on paper AWS Lambda only supports certain languages, it’s possible to deploy your code by uploading a ZIP file, which can contain anything you want — including arbitrary executables. All we need to do is make sure that those executables will execute properly under Lambda, and then call them from a Python script.

Thankfully, Amazon provides a Linux machine image of what the Lambda execution environment looks like, which we can compile our program against to produce our executable file. We’ll use a small example program in Haskell that writes any input it gets to a PostgreSQL database to show how to do this. Don’t worry if you don’t know any Haskell — the program’s pretty basic, and I’ll explain syntax as we go.

We start off by launching an EC2 based on the above AMI. Make sure to pick at least a t2.medium or you might have trouble compiling due to lack of memory.

Connect to the instance through SSH. We’ll use Stack to compile our Haskell code. Install it on the instance using the instructions on Stack’s website: curl -sSL https://get.haskellstack.org/ | sh . Install a few extra dependencies we’ll need using sudo yum install gcc make libffi zlib-devel gmp-devel. Finally, set up a new project using stack new lambda-example . I personally prefer to write and check in code directly from the instance, but you might prefer to get code onto the EC2 using whatever version control you’re using.

Edit lambda-example/app/Main.hs and replace it with the following code:

In a nutshell: read all the data on stdin and insert it into the database. Get the connection information from environment variables. We use (?) inside the SQL query to perform parameter substitution, with escaping handled for us.

Add postgresql-simple to the list of dependencies for the executable inside lambda-example/package.yaml, and compile the project using stack build (which will take a while the first time you run it). When we try to compile this, we run into a small problem: we don’t have the C libraries for connecting to PostgreSQL installed on our EC2. We can quickly fix this with a sudo yum install postgresql-devel. Once it’s built, you can test it by setting the appropriate environment variables and running stack exec lambda-example-exe.

Yay, we have data in our database!

So far, so good! All we need to do now is to package the whole thing up into a ZIP file and call into it from a Python script. Here’s a bare-bones Python script to do just that:

You can copy the compiled executable by running stack exec -- which lambda-example-exe | xargs -i cp {} ./lambda-handler. Zip together the compiled executable and the Python script into the following structure:

lambda-deployment-package.zip
├── lambda-handler.py
└── lambda-handler  # the compiled Haskell executable

Finally, download it from your EC2 using sftp, upload the whole thing to Lambda, and set the handler name accordingly:

When we test this, though, we run into a problem.

Even though it ran fine on our EC2 instance, it breaks when we try to run it on AWS Lambda. Why? Remember that we ran into an issue when building and had to install postgresql-devel? The Lambda is missing those dependencies as well, and our Haskell program is erroring out when it tries to find them at runtime.

Basically, a Linux executable can have two kinds of code dependencies: static and dynamic. Static dependencies get compiled directly into the executable file, but dynamic dependencies don’t, and are instead looked up in the filesystem at runtime. Here, our dynamic dependencies don’t exist in the filesystem, because we only copied the executable itself onto the Lambda function.

Thankfully, we have a way to solve this: copy the dynamic dependencies into our ZIP file, and patch the executable so that it looks in the current directory for dynamic dependencies as well¹.

Run ldd on your executable and you’ll see a list of all the dynamic dependencies it has, as well as where they’re currently located on the filesystem. We’ll copy all of these into a directory called dynlibs, then use PatchELF to redirect where the executable searches for them. Download PatchELF and compile it in the standard C way, with a ./configure && make && make install, then edit the executable, setting its RPATH to $ORIGIN/dynlibs.

We do a little bit of text manipulation to pull just the filenames from ldd’s output and copy them. Running ldd again on our patched executable, we can see that the missing file libpq.so.5 is now being found inside our local directory!

With this, our full program can finally run on AWS Lambda. Zip everything back up again, into the following structure:

lambda-deployment-package.zip
├── lambda-handler.py
├── lambda-handler
└── dynlibs
    ├── libcom_err.so.2
    ├── libcrypto.so.10
    ├── libcrypt.so.1
    ├── libc.so.6
    ...
    └── libz.so.1

Upload it again, and test it.

Finally — success!

This was just a small example that accessed a database, but you now have enough tools to build as complicated of a Lambda function as you want. Plus, this doesn’t just work for Haskell — the steps shown here work for any Linux executable, so you’re free to run Rust, C, or whatever you else you want on AWS Lambda.

Here’s a Makefile for our lambda-example project that you can adapt for your own purposes. It provides both build and package targets to easily create your deployment package. I’ve also duplicated the Python wrapper for your convenience.

Happy hacking!

Found this article useful? Want to work with awesome people to fight the $40 billion payday loan industry and take back payday? Earnin is hiring!

Footnotes

[1] We could also do this by statically compiling our Haskell executable, thereby removing all its dynamic dependencies, but I’ve found that patching the RPATH is easier and results in less weird errors down the line.

Photo by Cristina Cerda on Unsplash