Playing with Perl-based Lambda functions

Like any good Unix sysadmin, I used to spend a lot of my time writing Perl scripts. (Yes, that’s how old I am — I still say Unix rather than Linux). Of course, Perl was not just for sysadmin tasks — I’ve written many complete applications using it and for a long time, it was my language of choice.

We were joking at work the other day that it would be interesting to be able to use Perl to write AWS Lambda functions. So I Googled it and found this blog post — AWS custom runtime for lambda really works: How I developed a lambda in Perl — which gives a really nice overview of how to use the newer features of Lambda (specifically, layers and custom runtimes) to allow the use of Perl for writing Lambda functions.

Neat.

However, a few things are slightly glossed over in that blog post and, in any case, it doesn’t include the necessary modules to allow you to create Perl functions that interact with other AWS services, so I thought I’d have a go at building a Perl runtime for myself.

It’s actually pretty simple. And, even if you have no interest in Perl (which I guess is most people these days), what you learn will be useful for creating other Lambda custom runtimes.

Lambda layers are added by uploading a Zip file. The easiest way to create a Zip file with everything in the right place is to build what you want using an Amazon Linux EC2 instance, then grab the bits you need for your layer.

So, to create your new Perl custom runtime, start by creating a new Amazon Linux EC2 instance. Log into it and install the Linux development tools and the ActiveState Perl distribution, as follows:

$ sudo yum -y update
$ sudo yum groupinstall “Development Tools”
$ cd /tmp
$ wget https://downloads.activestate.com/ActivePerl/releases/5.24.3.2404/ActivePerl-5.24.3.2404-x86_64-linux-glibc-2.15-404865.tar.gz
$ zcat ActivePerl-5.24.3.2404-x86_64-linux-glibc-2.15-404865.tar.gz | tar xvf -
$ cd ActivePerl-5.24.3.2404-x86_64-linux-glibc-2.15-404865
$ sudo ./install.sh

Make sure you enter /opt/perl as the destination directory of the ActiveState install when prompted.

We are now almost ready to create the Zip files holding our Lambda layers. First though, we want to add some additional Perl libraries so that we can interact with other AWS services using our Perl-based Lambda functions. We do this by installing the Paws modules from CPAN, as follows:

$ sudo /opt/perl/bin/cpan -y install Paws
$ sudo /opt/perl/bin/cpan -y install Data::Printer
$ cd /opt/perl
$ sudo rm -rf html man
$ cd site
$ sudo rm -rf html man

Note that we are using the version of cpan that was created as part of the ActiveState installation above.

Data::Printer is installed because it is useful for debugging the output from calls to various AWS services. The html and man directories are removed to save space.

Finally, we create a symlink from /opt/bin/perl to /opt/perl/bin/perl (just to make sure that the right version of Perl is being used by default):

$ ln -s /opt/perl/bin/perl /opt/bin/perl

Layers are deployed by uploading a Zip file to AWS Lambda. Remember that when you create layers, they are going to be installed under /opt on the target Lambda host. That’s why we have installed everything under /opt here. Because of this, you need to change your working directory to /opt before creating the Zip file. Also remember that the maximum upload size for each layer is 50M. Unfortunately, by installing Paws, we have taken ourselves over that limit. Therefore, we need to deploy our work so far using two separate layers: one for the main Perl distribution and one for the Paws modules. This has the added advantage of giving us the option of using only the Perl layer if we don’t need Paws.

$ cd /opt
$ zip -r /tmp/perl.zip perl bin/perl -x perl/site/lib/Paws/\* -x perl/site/lib/Paws.pm
$ zip -r /tmp/paws.zip perl/site/lib/Paws perl/site/lib/Paws.pm

Now we have two Zip files, perl.zip and paws.zip, ready to be deployed.

Next, we need to configure the AWS CLI so that we can publish the layers to AWS.

$ aws configure

Type in your preferred AWS access key id, secret access key, default region name and default output format.

Then you are ready to deploy your layers:

$ aws lambda publish-layer-version --layer-name perl --zip-file fileb:///tmp/perl.zip
$ aws lambda publish-layer-version --layer-name paws --zip-file fileb:///tmp/paws.zip

Remember that each time you publish a layer, you get a new version and you will need to use the ARN of that version in any future Lambda deployments.

As described in the original blog post, we also need a third layer containing the bootstrap file (this is the first thing that is run as part of a custom Lambda environment) and a small Perl template file (which is what calls the Perl script that you ultimately deploy as your Lambda function). Read the blog post to understand how these work to create the custom runtime.

These files should be created directly in /opt. Copy and paste the content from the original blog post (or see below). Then deploy them as a third layer, as follows:

$ cd /opt
$ sudo vi bootstrap
$ sudo vi perl_runtime.pl.tpl
$ chmod 755 bootstrap perl_runtime.pl.tpl
$ zip -r /tmp/perl-runtime.zip bootstrap perl_runtime.pl.tpl
$ aws lambda publish-layer-version --layer-name perl-runtime --zip-file fileb:///tmp/perl-runtime.zip

For completeness, I’ll copy the code of these two files below:

bootstrap
#!/bin/sh
set -euo pipefail
# Initialization - load function handler
# The function handler is of the format [handler-file-name-withon-extn].[handler-method]
# The handler file/script name
lambda_script_name="$(echo $_HANDLER | cut -d. -f1)"
# The handler method/function name
handler_name="$(echo $_HANDLER | cut -d. -f2)"
# Processing
while true
do
  HEADERS="$(mktemp)"
# Get an event
  EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
  REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
cd /tmp
# The directory location $LAMBDA_TASK_ROOT which is /var/task is readonly,
  # So copy the perl_runtime.pl.tpl file from layer location /opt/ to /tmp
  cp /opt/perl_runtime.pl.tpl perl_runtime.pl
# Copy the perl lambda script from LAMBDA_TASK_ROOT
  cp $LAMBDA_TASK_ROOT/$lambda_script_name.pl $lambda_script_name.pl
# Update the perl runtime file with the actual script name and
  sed -i "s/PLACEHOLDER_SCRIPT_NAME/$lambda_script_name/g" perl_runtime.pl
  sed -i "s/PLACEHOLDER_HANDLER_NAME/$handler_name/g" perl_runtime.pl
# Execute the handler function from the script
  export PERL5LIB=/opt/perl/lib
  RESPONSE=$(/opt/perl/bin/perl perl_runtime.pl "$EVENT_DATA")
# Send the response
  curl -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response"  -d "$RESPONSE"
done
perl_runtime.pl.tpl
#!/usr/bin/perl
use strict;
my $lambda_script_name = 'PLACEHOLDER_SCRIPT_NAME.pl';
require "$lambda_script_name";
my $handler_return_value=PLACEHOLDER_HANDLER_NAME(@ARGV);
print $handler_return_value

You now have 3 layers which can be used to support the deployment of Lambda functions written in Perl.

The original blog post shows how to deploy a test function using the CLI. Instead, let’s go to the AWS console and deploy one from there. There are some example scripts at https://github.com/pplu/aws-sdk-perl/tree/master/examples but I also provide one below — a simple script to list your EC2 instances and return a list of their identifiers in a JSON structure.

Go to the AWS console. Create a new Lambda function. Select the Author From Scratch option. Choose Custom Runtime and select or create a Role — for the script below, choose Simple Microservice Permissions but you’ll also need to add EC2 Read-only permissions as well (do this after the function has been created). Hit Create function.

Because you selected Custom Runtime, the function automatically gets created with bootstrap and hello.sh files. You don’t need these… so delete them.

Now add the three layers we created above — perl, paws and perl-runtime (I don’t think the order matters). Add them using their ARNs (and remember to choose the latest version if you have more than one).

Now create a Perl script called lambda_script.pl with the following content:

lambda_script.pl
use strict;
use warnings;
use Paws;
use JSON;
my $result;
my @instances;
sub handler {
  #my $event = $_[0];
  #my $response = decode_json($event);
  my $response = {};
  my $ec2 = Paws->service(‘EC2’, region => ‘eu-west-1’);
  my $result = $ec2->DescribeInstances;
  my $r;
  foreach $r (@{$result->Reservations}) {
    #print “Reservation: “.$r->ReservationId.”\n”;
    my $i;
    foreach $i ( @{$r->Instances}) {
      #print “Instance: “.$i->InstanceId.”\n”;
      my $v = {
        Id => $i->InstanceId
      };
      push @instances, $v;
    }
  }
  $response->{‘Instance’} = \@instances;
  return encode_json($response)
}
1;

Click Save and then Test. You should see a list of EC2 instance ids in a JSON structure.

OK, we’re done.

The point here was not to recommend writing all your Lambda functions in Perl! Instead, the point was to understand a bit more about how layers and the custom runtime mechanism works. For what it’s worth, the performance of Perl isn’t great and, interestingly, it gets significantly worse as soon as you add the ‘use Paws;’ line to your lambda function code — like about 5 or 6 times slower.

But hopefully, what we’ve learned above will be helpful in understanding how Lambda layers and custom runtimes work.

For background reading, see:

• AWS Lambda Layers
• Custom AWS Lambda Runtimes
• Lambda Layers and Custom Runtimes