AWS custom runtime for lambda really works: How I developed a lambda in Perl

At re:Invent 2018, AWS announced a new lambda feature called custom runtime for lambda and what exactly it means is you can develop lambda functions in not only the out-of-box supported runtimes like Java, C#, Python etc but also using any other programming language.

So I decided to give it a try and guess what its worked as promised. I am able to develop and deploy a lambda function using Perl (no kidding!!).

Lets quickly review the key points for lambda custom runtimes and then we will go through some real code to try it out.

  • The runtime is responsible to execute the lambda function’s code, it reads the handler info (file name and the method name) from the pre-defined environment variable _HANDLER
  • The runtime passes the event data to the function handler, and posts the response from the handler back to Lambda.
  • The custom runtime runs on the standard lambda execution environment, so it will be your responsibility to bring along any library or language support you need for the custom runtime.
  • Make sure the lambda Runtime is as below on AWS console or “Provided” in the create-function lambda CLI.
  • The runtime can be included in your function’s deployment package, or in a layer.
  • A custom runtime’s entry point is an executable file named as “bootstrap”, it should be in the lambda deployment package or in a referenced lambda layer

Key environment variables that you should be using in the bootstrap file to initialize and executethe lambda task:

  • _HANDLER: The format is [file-name].[method], the method in this file will be executed
  • LAMBDA_TASK_ROOT: This is where on lambda execution environment the deployment package files will be available. The value is “/var/task” and its a read-only location

I have mentioned a few of the key steps in bootstrap executable file below, for additional details refer here:

  • Initialize the function — Load the handler file.
  • Handle errors — If an error occurs, call the initialization error API and exit immediately.
  • Get an event — Call the next invocation API to get the next event. The response body contains the event data. Response headers contain the request ID and other information.
  • Create a context object — Create an object with context information from environment variables and headers in the API response.
  • Invoke the function handler — Pass the event and context object to the handler.
  • Handle the response — Call the invocation response API to post the response from the handler.

Lets deep-dive on how we can develop a lambda using perl runtime:

A lambda function code in perl needs the perl interpreter and the dependant libraries.

  • I downloaded the ActivePerl community edition for linux (x86_64) from here
  • I created a lambda layer using the download file, you cannot upload it as-is due to size limitation, lambda layer only supports a zip file upto 50MB so to keep the size within limit I had to delete the man and html directories, the final zip file only has below directories:
  • The lambda layer looks as below:

Let's take a quick look into the bootstrap file:

  • Its a shell script
#!/bin/sh
set -euo pipefail
# Initialization - load function handler
# The function handler is of the format [handler-file-name-withon-extn].[handler-method]
# The handler file/script name
lambda_script_name="$(echo $_HANDLER | cut -d. -f1)"
# The handler method/function name
handler_name="$(echo $_HANDLER | cut -d. -f2)"
# Processing
while true
do
HEADERS="$(mktemp)"
  # Get an event
EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
cd /tmp
# The directory location $LAMBDA_TASK_ROOT which is /var/task is readonly,
# So copy the perl_runtime.pl.tpl file from layer location /opt/ to /tmp
cp /opt/perl_runtime.pl.tpl perl_runtime.pl
# Copy the perl lambda script from LAMBDA_TASK_ROOT
cp $LAMBDA_TASK_ROOT/$lambda_script_name.pl $lambda_script_name.pl
# Update the perl runtime file with the actual script name and
sed -i "s/PLACEHOLDER_SCRIPT_NAME/$lambda_script_name/g" perl_runtime.pl
sed -i "s/PLACEHOLDER_HANDLER_NAME/$handler_name/g" perl_runtime.pl
# Execute the handler function from the script
export PERL5LIB=/opt/perl/lib
RESPONSE=$(/opt/perl/bin/perl perl_runtime.pl "$EVENT_DATA")
# Send the response
curl -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response" -d "$RESPONSE"
done
  • Extract the lambda handler file name and the handler method from the environment variable _HANDLER
  • Run an infinite loop to handle incoming requests
  • Get the EVENT_DATA from the lambda runtime API: http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next
  • Get the REQUEST_ID from the HEADERS
  • I have a template perl script perl_runtime.pl.tpl that acts as a glue code between the bootstrap and the actual lambda handler file, I have kept it so that I can create a perl script with the actual handler file name and the function name.
#!/usr/bin/perl
use strict;
my $lambda_script_name = 'PLACEHOLDER_SCRIPT_NAME.pl';
require "$lambda_script_name";
my $handler_return_value=PLACEHOLDER_HANDLER_NAME(@ARGV);
print $handler_return_value
  • The bootstrap executable file and the perl_runtime.pl.tpl file is in a separate lambda layer named as perl-runtime:
  • The bootstrap file and perl_runtime.pl.tpl are already in PATH and available under “/opt/”
  • The bootstrap executable file copy over the perl_runtime.pl.tpl file to /tmp and replaces the placeholder variable with actual values and create the actual perl_runtime.pl
  • The bootstrap executable also copy over the actual lambda handler file to /tmp so that it can be invoked by the perl_runtime.pl
  • As we already have perl interpreter as runtime and the lambda will refer it, so the interpreter and the libraries are already available under “/opt/perl” and the bootstrap executable using them as below to involve the perl_runtime.pl script:
export PERL5LIB=/opt/perl/lib
RESPONSE=$(/opt/perl/bin/perl perl_runtime.pl "$EVENT_DATA")

Lambda Handler Script:

The lambda handler script file “lambda_script.pl” just read the incoming json event and add a reply field and return back the same.

use strict;
use warnings;
use JSON;
sub handler {
my $event = $_[0];
my $reponse = decode_json($event);
$reponse->{'reply'} = "Thanks AWS custom lambda runtime!! Now even perl can be a runtime!!";
return encode_json($reponse)
}
1;

The actual execution flow:


Lambda function deployment:

  • Create perl interpreter lambda layer:

This is the perl interpreter and the library zip file.

aws lambda publish-layer-version --layer-name perl --zip-file fileb://perl.zip
  • Create perl-runtime lambda layer:

It has the bootstrap executable shell script and perl_runtime.pl.tpl files

aws lambda publish-layer-version --layer-name perl-runtime --zip-file fileb://perl-runtime.zip
  • Create the lambda function:

The perl-lambda-poc.zip file contains the lambda_script.pl file, the actual lambda function handler.

aws lambda create-function --function-name perl-lambda-poc \
--zip-file fileb://perl-lambda-poc.zip \
--handler lambda_script.handler \
--runtime provided \
--role arn:aws:iam::1234554321:role/my-lambda-role \
--layers perl perl-runtime

You can invoke the lambda function with any json payload, I tested it with:

{
"request": "Hey perl lambda!!",
}

And the response came back as:

{
"request": "Hey perl lambda!!",
"reply": "Thanks AWS custom lambda runtime!! Now even perl can be a runtime!!"
}