Automated: AWS Lambda Layer for Sharing Java Code

A fully automated approach using Maven for sharing Java code in Lambda Functions via Lambda Layers

Published in

The Startup

14 min readSep 13, 2020

Few years ago, I bought into the Smart Home craze and ventured down Amazon’s Echo path. Now with numerous smart devices scattered around the house (and car), I’ve a need for few “skills” to further streamline the automation.

Note: It is assumed that you have some level of familiarity with AWS technologies and terminologies as well as a good understanding of Java and Maven.

Before venturing down the “build my own skill” (using AWS Lambda) path, I wanted to set the ground work in place. In doing so, I came across a challenge that took more effort than I liked, to overcome.

As the title says, it’s about how to share Java code using Lambda Layers in an automated fashion. Simply, when I commit common code into the repository, it should be available as a Lambda Layer after a few minutes. Then, in whichever Lambda Function that I need to use the common code, all I want to do is point it to the Lambda Layer (version) and commit that change into the repository. Similar to the Lambda Layer, the Lambda Function should be available to handle requests after a few minutes.

Only time I want to touch the AWS console is to check where things are and their corresponding statuses or errors.

In the industry, this approach is called CI/CD. However, the tricky part is not the CI/CD but rather how the Java code is structured to provide the expected outcome, i.e. Automated way (through CI/CD) to share Java code.

Motivation

When I started on this exercise, most of the resources I came across in the wild world of internet were describing how to do using the AWS Console or having additional manual steps in between or even using sam build (refer Extras below). Hence the reason, I ventured into finding an automated approach. Though in this example I've used scripts quite a bit, it is to perform cloudformation deploy of stacks with relevant parameter values. Which if you are familiar, is ideally done once or very infrequently in comparision to writing implementation Java code (which gets delivered automatically by CI/CD).

Getting Started

The full source of this example is available in my GitHub repo p13i-eg-jl

So, what do you need to get started?

A GitHub account — you can use any Git repository, but I’ve used GitHub because with private repos and Packages offering, it doubles as both the code repo as well as the artefact repo.

Create a private repo named mvn-repo — this is the placeholder repo that will act as your Packages repository. You can change this later.
Create a GitHub token having the following permissions

repo:* (all) to be able to manage code
write:packages, read:packages and delete:packages to be able to manage Packages
admin:repo_hook to be able to create and delete web hooks from AWS (refer Extras below)

An AWS Account — obviously

Create a Secret in AWS Secrets Manager as shown here, where the secret-string is {"username":"<github-username>","token":"<token-generated-above>"}. Make sure that the Secret is created in the region where you would be setting up the stack (using your local setup mentioned below)

A comprehensive development environment — may it be local or virtual, as long as it has all the tooling like your git set up to push code, AWS CLI set up with appropriate IAM user, Java, Maven, an IDE, etc.

TL;DR

Some of us just want to get to the action and then learn of how we got there. In the interest of such spirited souls, here are the steps of the shortest possible path to the end state, i.e. having a Lambda Function using Lambda Layer.

Clone the above repositories using git clone --recurse-submodules https://github.com/rajivmb/p13i-eg-jl.git. To make it easier for this example, I've used git submodules to combine both the Lambda Layer and Lambda Function into a single repo. Hence the need for --recurse-submodules in the clone command.
Change into the checked out directory cd p13i-eg-jl.
Execute the setup script ./setup.sh and follow the prompts. You will have to pass in your git repo and AWS Secret created in the Getting Started above, since the defaults are that of mine.
Wait for the setup process to complete. It should roughly take about 15 minutes to complete.

Now you have a Lambda Function written in Java sharing common code using Lambda Layer, all delivered via CI/CD automatically.

You should see a similar output to the following from the setup process

$ ./setup.sh 

Setting up Lambda Layer
***********************
Initialising...
DIR is p13i-eg-jpll
Commencing setup...
Enter component name [P13i-Eg-JPLL] or press <Enter> to accept default, you have 30s: 
Enter your Secret name of GitHub token stored in AWS Secrets Manager [P13iAWSGitHubTokenSecret], you have 30s: 
Enter your GitHub Packages (repo) URL to use as private Maven repo [https://maven.pkg.github.com/rajivmb/p13i-mvn-repo], you have 30s: 
Starting to setup P13i-Eg-JPLL
Deploying stack of P13i-Eg-JPLL

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - P13i-Eg-JPLL
Deploying Lambda Layer via Pipeline: P13i-Eg-JPLL-Pipeline-<suffix>
Pipeline status is: InProgress. Waiting... | [ 04m 15s ]
Pipeline status is: Succeeded
Completed setup of P13i-Eg-JPLL

Lambda Layer setup completed in           [ 08m 22s ]

Setting up Lambda Function
**************************
Initialising...
DIR is p13i-eg-jglf
Commencing setup...
Enter component name [P13i-Eg-JGLF] or press <Enter> to accept default, you have 30s: 
Enter your Secret name of GitHub token stored in AWS Secrets Manager [P13iAWSGitHubTokenSecret], you have 30s: 
Enter your GitHub Packages (repo) URL to use as private Maven repo [https://maven.pkg.github.com/rajivmb/p13i-mvn-repo], you have 30s: 
Fetching resource P13iMITEgJavaParentLambdaLayerArn from P13i-Eg-JPLL-DEPLOY outputs
Fetching latest version of Lambda Layer: arn:aws:lambda:ap-southeast-2:<AWS::AccountId>:layer:P13i-Eg-JPLL-Layer.
Latest version of Lambda Layer: arn:aws:lambda:ap-southeast-2:<AWS::AccountId>:layer:P13i-Eg-JPLL-Layer. is 3
Starting to setup P13i-Eg-JGLF
Deploying stack of P13i-Eg-JGLF

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - P13i-Eg-JGLF
Deploying Lambda Function via Pipeline: P13i-Eg-JGLF-Pipeline-<suffix>
Pipeline status is: InProgress. Waiting... \ [ 03m 30s ]
Pipeline status is: Succeeded
Completed setup of P13i-Eg-JGLF
./setup.sh: line 17: up.sh: command not found

Lambda Function setup completed in        [ 07m 21s ]

Invoking Lambda Function
************************

DIR is p13i-eg-jglf
Enter component name [P13i-Eg-JGLF] or press <Enter> to accept default, you have 30s: 
Enter greeting name [github.com/rajivmb], you have 30s: 
{
    "ExecutedVersion": "$LATEST", 
    "StatusCode": 200
}
"Hello github.com/rajivmb, greetings from Java Lambda Function using Lambda Layer"


Lambda Layer setup completed in           [ 08m 22s ]
Lambda Function setup completed in        [ 07m 21s ]
Setup completed in                        [ 16m 49s ]

Note: If you executed the TL;DR version multiple time (for whatever reasons) and changed your private GitHub repository that you are using as your private Maven repository between each execution, then you will encounter the following build failure during the deploy goal of p13i-eg-jpll project.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project p13i-java-parent-example: Failed to deploy artifacts: Could not transfer artifact com.p13i.mit.aws.example:p13i-java-parent-example:jar:1.0.0-20200913.122552-1 from/to internal.repo (https://maven.pkg.github.com/***/mvn-repo): Transfer failed for https://maven.pkg.github.com/***/mvn-repo/com/p13i/mit/aws/example/p13i-java-parent-example/1.0.0-SNAPSHOT/p13i-java-parent-example-1.0.0-20200913.122552-1.jar 422 Unprocessable Entity -> [Help 1]

The key thing to note here is error code, i.e. 422 Unprocessable Entity. You get this error, because you probably still have the same artefact stored in the repository that you had used in previous executions as opposed to the one you are using in the current execution. Once you delete the artefact from the old repository or use that same repository, then you would not encounter this error.

The Layer Project Explained

Layer project is in the subdirectory p13i-eg-jpll. Alternately you can clone the repo p13i-eg-jpll

The key focus of this post is about how to set up the Java project correctly to function as a Lambda Layer. Before getting into that, lets first establish what “common code” we intend to share. There are two flavors of code that are sharable;

Dependencies — these are the artefacts we reference in the <dependencies> section of the POM
Java code that we write — like utility classes, model classes, etc.

In essence, #2 is actually #1. Because, when you have some code that you want to share between your Java implementations, you create an artefact of it and then add it as a dependency on the implementations where you want to use it.

So, the approach here is the same. Declare everything as a dependency, so that they are packaged as articulated in AWS documentation, where for Java, the shared code is expected to be placed in the java/lib directory.

Project Structure

The project is a multi-module Maven project comprising of the following modules

Source — this is the module where you will place all the dependencies along with your own Java classes. For example, I’ve place the FunctionResourceBundle class in here.
LambdaLayer — this module is purely for Lambda Layer packaging purposes only. This will have a dependency on the artefact generated by the Source module above.

Project Packaging

If you inspect the three POM files i.e. pom.xml, Source/pom.xml and LambdaLayer/pom.xml you will notice that both pom.xml and LambdaLayer/pom.xml have the property maven.deploy.skip set to true. This is because, the build process ( buildspec.yaml) performs the deploy goal. In doing so, we only want the Source artefact to be uploaded to our internal repo (GitHub Packages), so that it can be shared by Java implementation with a provided scope (as you will see in the Lambda Function below)

The LambdaLayer/pom.xml only has a single dependency, i.e. the Source artefact, since this module is what will be built to satisfy Lambda Layer packaging requirements by the build process and subsequently deployed by the pipeline. The Shade plugin is configured as follows, to match the Lambda Layer packaging requirements mentioned earlier.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>3.1.1</version>
    <executions>
        <execution>
            <id>copy-dependencies</id>
            <phase>prepare-package</phase>
            <goals>
                <goal>copy-dependencies</goal>
            </goals>
            <configuration>
                <outputDirectory>
                    ${project.build.directory}/classes/java/lib
                </outputDirectory>
                <includeScope>runtime</includeScope>
            </configuration>
        </execution>
    </executions>
</plugin>

Here prepare-package phase performs the copy-dependencies goal that copies over all of the dependeny jars into the java/lib directory in the classpath.

At the end of a successful deployment, you should see something similar to the following in your AWS Console.

**Lambda Layer as displayed in the AWS Lambda console under Layers.**

**Artefact file as displayed in the S3 bucket used by CodeBuild.**

Take note of the file size. The packaged Lambda Layer artefact is 21.0 MB. We will compare this with that of the Lambda Function below.

The Function Project Explained

Function project is in the subdirectory p13i-eg-jglf. Alternately you can clone the repo p13i-eg-jglf

This is a normal Lambda Function implementation, nothing fancy here except how the dependency is managed. If you inspect the pom.xml file you will see the dependency on the parent artefact declared as follows.

<dependency>
    <groupId>com.p13i.mit.aws.example</groupId>
    <artifactId>p13i-java-parent-example</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <scope>provided</scope>
</dependency>

Key thing to note here is the <scope> having the value provided. This tells Maven not to package this dependency in to the build artefact because it will be provided at run time. Refer Introduction to the Dependency Mechanism in Maven documentation. Also note that the dependency is on the Source artifact, not on LambdaLayer. Because in a conventional Maven Java project, you would not care about the bespoke packaging requirements enforced by the runtime, i.e. AWS Lambda in this case.

Function Implementation

The implementation is a simple “Hello World”. If a name parameter was passed then the greeting would be for the value of the parameter. The demonstrated point here is that, the greeting message is constructed using the FunctionResourceBundle class that we packaged in the Lambda Layer.

If you didn’t do the TL;DR approach above, you can deploy the Lambda Function by running the setup.sh script. Thereafter, you can invoke the Lambda Function by running the invoke.sh script. Assuming everything went to plan, you should see the following output (if you accepted the defaults). You should also see something similar to the image following the output in your AWS Console.

{
    "ExecutedVersion": "$LATEST", 
    "StatusCode": 200
}
"Hello github.com/rajivmb, greetings from Java Lambda Function using Lambda Layer"

**Lambda Function as displayed in the AWS Lambda console under Functions.**

**Lambda Function showing the Lambda Layer attached.**

Take note of the file size yet again. The packaged Lambda Function artefact is just 4.5 KB as opposed to the Lambda Layer above, which is 21.0 MB. This is proof that the use of provided scope worked as expected in not packaging the dependency into the Function artefact.

The fact that this Lambda Function is able to execute is yet another confirmation that the Lambda dependencies are all being provided to the Lambda Function by the Lambda Layer at run time and therefore we don’t have to individually package them into every Lambda Function.

Extras

GitHub Workflow to purge and (re)deploy

When using Maven 3, each SNAPSHOT version you deploy gets deployed as a new artifact, versioned with the timestamp. Refer Maven Deploy Plugin for more details. For ease of reference, here is the excerpt.

Major Version Upgrade to version 3.0.0
Please note that the following parameter has been completely removed from the plugin configuration:
* uniqueVersion
As of Maven 3, snapshot artifacts will always be deployed using a timestamped version.

Unlike the purpose built Maven artefact repos like Nexus or Artifactory, GitHub Packages does not offer a centralised configuration to prune or limit the number of SNAPSHOT versions to retain. In order to optimise the storage utilisation, as a workaround a GitHub Workflow is placed in p13i-eg-jpll repo to purge and deploy (similar to what the buildspec.yaml does) once a week.

Note: The workflow uses a separate token than the one that is available by default. As it states on Authenticating with the GITHUB_TOKEN, this token only has permissions to the repo where the workflow is. For ease of reference, here is the excerpt.

The token’s permissions are limited to the repository that contains your workflow.

Because I’m using a separate repository as my internal Maven repo, a new token with relevant permissions had to be created to be used in the workflow.

Use of `cloudformation` instead of `sam`

Though I started off using sam build and sam package, I switched to using mvn package/ mvn deploy and cloudformation deploy because the sam build didn't seems to honour the provided scope for dependencies. After performing sam build you can see all the dependencies in the build output directory as shown below.

**Local build output directory of SAM build.**

Similarly, after performing sam package you can see the size of the artifact is larger than it should be.

**Artefact file as displayed in the S3 bucket resulting from SAM Packageing.**

Therefore, you may get a false sense of using the Lambda Layer, while actually packaging your dependencies with your Lambda Function.

ImportValue and externalised Lambda Layer version

If you inspect the package.yaml file in p13i-eg-jglf, you can see the use of Fn::ImportValue and the parameterised LambdaLayerVersion. I used this purely for demonstration purposes. You are better off specifying the Lambda Layer dependency in the Function's template file explicitly. It is a more scalable approach and you have flexibility to use more than one Layer without complicating your implementation.

Disabled GitHub Webhook creation

Because you are running this as an example from my repo as the source, the P13iEgJPLLGitHubWebhook and P13iEgJGLFGitHubWebhook resources are disabled by default. If you were to fork my example repos, you can enable these resource creations by uncommenting the CreateGitHubWebHook parameter overrides in the setup.sh script.

Conclusion

No guide is complete without providing a teardown. Assuming you are already in the p13i-eg-jl directory, execute the teardown script ./teardown.sh and follow the prompts.

You should see a similar output to the following from the teardown process

$ ./teardown.sh 

Tearing down Lambda Function
****************************

Initialising...
DIR is p13i-eg-jglf
Commencing tear down...
Enter component name [P13i-Eg-JGLF] or press <Enter> to accept default, you have 30s: 
Starting to tear down P13i-Eg-JGLF
Fetching resource BuildArtifactsBucketName from P13i-Eg-JGLF outputs
Deleting stack arn:aws:cloudformation:ap-southeast-2:<AWS::AccountId>:stack/P13i-Eg-JGLF-DEPLOY/<stackId>
Stack status is: DELETE_IN_PROGRESS. Waiting... / [ 00m 00s ]
Stack status is: DELETE_COMPLETE.
Deleting stack arn:aws:cloudformation:ap-southeast-2:<AWS::AccountId>:stack/P13i-Eg-JGLF/<stackId>
Stack status is: DELETE_IN_PROGRESS. Waiting... \ [ 00m 05s ]
Stack status is: DELETE_COMPLETE.

Deleting S3 bucket : p13i-eg-jglf-buildartifactsbucket-<suffix>.
delete: s3://p13i-eg-jglf-buildartifactsbucket-<suffix>/P13i-Eg-JGLF-Artifacts/d8d7ab279f9e965e06a22ac57a877b8a
delete: s3://p13i-eg-jglf-buildartifactsbucket-<suffix>/P13i-Eg-JGLF-Pip/BuildArtif/B2seD5N
delete: s3://p13i-eg-jglf-buildartifactsbucket-<suffix>/P13i-Eg-JGLF-Pip/SourceCode/aq19ktb.zip
delete: s3://p13i-eg-jglf-buildartifactsbucket-<suffix>/codebuild-cache/5b533265-9019-44ef-b9ef-2a3edc17cfc9
remove_bucket: p13i-eg-jglf-buildartifactsbucket-<suffix>
Completed tearing down P13i-Eg-JGLF

Lambda Function teardown completed in           [ 01m 02s ]

Tearing down Lambda Layer
*************************

Initialising...
DIR is p13i-eg-jpllCommencing tear down...
Enter component name [P13i-Eg-JPLL] or press <Enter> to accept default, you have 30s: 
Starting to tear down P13i-Eg-JPLL
Fetching resource BuildArtifactsBucketName from P13i-Eg-JPLL outputs
Fetching resource P13iMITEgJavaParentLambdaLayerArn from P13i-Eg-JPLL-DEPLOY outputs
Deleting stack arn:aws:cloudformation:ap-southeast-2:<AWS::AccountId>:stack/P13i-Eg-JPLL-DEPLOY/<stackId>
Stack status is: DELETE_IN_PROGRESS. Waiting... | [ 00m 00s ]
Stack status is: DELETE_COMPLETE.
Deleting stack arn:aws:cloudformation:ap-southeast-2:<AWS::AccountId>:stack/P13i-Eg-JPLL/<stackId>
Stack status is: DELETE_IN_PROGRESS. Waiting... - [ 00m 05s ]
Stack status is: DELETE_COMPLETE.

Deleting S3 bucket : p13i-eg-jpll-buildartifactsbucket-<suffix>.
delete: s3://p13i-eg-jpll-buildartifactsbucket-<suffix>/P13i-Eg-JPLL-Artifacts/5e7f0354aa9002befc9018c3576a7adf
delete: s3://p13i-eg-jpll-buildartifactsbucket-<suffix>/P13i-Eg-JPLL-Pip/BuildArtif/ohCQYOo
delete: s3://p13i-eg-jpll-buildartifactsbucket-<suffix>/P13i-Eg-JPLL-Pip/SourceCode/evx19yF.zip
delete: s3://p13i-eg-jpll-buildartifactsbucket-<suffix>/codebuild-cache/6316e8a5-7c7e-4898-a3ac-9b71b10efce8
remove_bucket: p13i-eg-jpll-buildartifactsbucket-<suffix>

Deleting Lambda Layer : arn:aws:lambda:ap-southeast-2:<AWS::AccountId>:layer:P13i-Eg-JPLL-Layer.
Deleting Layer Version 3
Completed tearing down P13i-Eg-JPLL

Lambda Layer teardown completed in              [ 01m 04s ]


Lambda Function teardown completed in           [ 01m 02s ]
Lambda Layer teardown completed in              [ 01m 04s ]
Teardown completed in                           [ 02m 06s ]

Note: The teardown process will not delete the AWS Secret you set up manually as part of Getting Started above. You will have to delete this yourself if you no longer need it.

Up Next

In this example, I’ve used git submodules to embed the Lambda Layer and the Lambda Function repos into an over arching example repo. As stated in the git submodules documentation (referenced above), using submodules for two way modification is not the best approach. A better approach would be to use a monorepository. In the next article, I will demonstrate my use of the monorepository concept using a practical implementation.

Acknowledgements

Acknowledging the inputs and feedback provided by Jamie Cansdale in my implementation of Git Workflow mentioned above.
Acknowledging the assistance rendered by my friend Srisaiyeegharan Kidnapillai in proofing this article and validating that the example not only “works on my machine” but on another person’s too.

References

Most references that assisted in writing this example are present in the code as comments prefixed with <Ref/>. They are as follows;