AWS Lambda and Custom Binaries

6 min readMar 28, 2020

It won’t be wrong to say that the advent of AWS and particularly AWS Lambda has changed the whole dynamics of application building. With Lambda, one just needs to have the application logic and leave the rest to AWS.

When working with Lambdas, even with such great services at their disposal, the developers at times are restrained in certain areas. One such is running custom binaries on lambdas and which this article tries to explain in nutshell.

Basic Terminologies

Before moving forward on this, let us understand the few terminologies, having a basic knowledge of which might come handy during the course of the article.

Binary File and Binary Package

A binary file is one that consists of compiled and executable code. For instance, compiling a basic “Hello World” C program using GCC involves building a binary which in turn writes output on the console.

A binary package is one that consists of contains (pre-built) executables. Usually, this is managed by a package manager such as brew on Mac-OS and apt-get on Debian Linux systems.

Shared Libraries

Shared Libraries are nothing but a set of libraries that can be associated with a program at runtime. There are a number of advantages of using shared libraries and covering them is beyond the scope of this article. Although, I would seriously recommend having a fair idea about why we use shared libraries in general.

There are many naming conventions when it comes to shared libraries and one such is as mentioned under.

“lib” followed by the library name, followed by “.so”, followed by “.” and a version number. For example : libpoppler.so.70

Essential Linux Commands

In order to have an unhindered reading experience, having some basic knowledge about the commands being used might come handy.

configure

There is a configure script associated with any package and when run, it checks if all the dependencies related to that particular package are pre-installed or not. This could be triggered by running the undermentioned command in the root directory of the package to be installed.

./configure

Apart from checking the dependencies, it is also responsible for the creation of Makefile which is discussed in the next section.

make and make install

Makefile usually contains steps that are responsible for compiling the source code into executable binaries. A typical Makefile consists of targets, prerequisites, and recipes.

make

The above command when executed, starts executing the targets specified in the Makefile.

Once we have successfully installed the package, the next step involves copying the binaries to appropriate locations from where they can be accessed. In most of the cases, the files are copied to one of the directories on the PATH variable.

make install

PATH and LD_LIBRARY_PATH

PATH is an environment variable that tells which directories to look for while searching executables.

Similarly, LD_LIBRARY_PATH is an environment variable that tells which directories to look for while searching shared libraries.

When and Why use custom binaries in lambda?

Although the Lambda runtime environment has a rich collection of pre-compiled binaries that are ready to use, there are few packages that require an altogether different set of binaries which leaves us with no option but to compile the binaries from the source first hand.

One such Python package is pdf2image. It is nothing but a wrapper around pdftoppm which is one of the tools from poppler-utils. These sets of tools are in turn built on the famous poppler library which is extensively used in pdf related operations.

To use pdf2image on lambda we need pdftoppm executable which by default is not available in lambda runtime kernel.

How to use custom binaries on lambda?

If we ever encounter a situation where we are required to use custom binaries on AWS Lambdas, we need to compile binaries along with the shared libraries on which those binaries depend. Let’s walk through it with the help of an example.

To successfully run the pdf2image into the lambda environment we will follow the undermentioned steps.

Compiling the source package on EC2 (or docker/local Linux environment).
Zipping and downloading the third-party binaries along with shared libraries.
Uploading package as a lambda layer.

Note:- There are many alternatives to each step that are mentioned in this article but the end result is the same and which is successfully running the third party binaries on our lambda.

Compiling source package

In this example, the compilation of the source code part will be undertaken on an EC2 instance. Although, this could have been easily done on any docker container and the end result for both is going to be the same i.e. we are going to get a set of executable binaries.

Installing build dependencies

There maybe be scenarios where the source code build process has few build dependencies and thus before initiating the build process, we need to install those dependencies.

In our example, the poppler source code has few build dependencies which need to be installed before we begin with the actual source code build process. Those dependencies can be installed using an undermentioned set of commands.

sudo apt-get install openjpeg-devel libjpeg-devel fontconfig-devel libtiff-devel libpng-devel xz gcc gcc-c++ epel-release zip cmake3

Downloading and building a source package

Once we have successfully, installed the build dependencies, the next step involves downloading the source package and triggering the build process.

In our case, the poppler source package can be downloaded and the build can be initiated using the undermentioned steps.

curl https://poppler.freedesktop.org/poppler-0.59.0.tar.xz | tar xJvcd poppler-0.59.0/ && ./configure --enable-static --enable-build-type=release && make && make install

Zipping and downloading the package

We have now successfully built our source code. The next step involves, handpicking the executables and their dependent shared libraries. We need to identify the shared libraries on which our binaries are dependent and this could be done using the undermentioned command.

ldd path/to/the/executable

In our case, the pdftoppm executable depends on the following set of shared libraries as listed in the image.

Once we have identified our shared libraries, the next step involves copying the required binaries and shared libraries into a package.

Although there are no restrictions on the directory structure, I prefer the undermentioned directory structure to have easy access to libraries and binaries.

.
+--package
|  +--bin
|  +--lib

Once we the skeleton of our directory ready with us, we can start copying the binaries and shared libraries into respective folders.

Note:- The executables and the directories should have read, write and execute permissions set on them to allow the lambda kernel to have full access to them. This is an important step, failing to which could result in “Access Denied” errors at lambda runtime. The files and folders could be given necessary permissions using the undermentioned set of commands.

chmod 755 /package/bin

Once we have our package ready we can zip it.

cd package
zip -r9 ../package.zip *

The next step involves downloading the package to local machine from our EC2 instance and this could be achieved using the below set of commands from the local machine.

scp -i path/to/key username@ec2ip:/path/to/file .

Uploading package to lambda layer

We are now done with the hard part and the only thing that remains is uploading our zip into a lambda layer and adding mandatory environment variables. Once that is done, we are all set to use our package.

Creating a Lambda Layer

There are many tutorials that greatly explain how layers work and how to use them. I seriously recommend going through any such tutorial before moving forward in this article.

The next critical step is to create two new environment variables and refer our binaries and shared libraries through them.