How to build pluggable Golang application and benefit from AWS Lambda Layers.

Golang — why is it worth your attention?

Golang is an open source programming language designed and implemented by Google. It’s very widely used in modern applications especially in the cloud. It’s the most characteristic features are:

• Golang is statically typed - it provides less flexibility, but protects you from making mistakes,
• It is not object-oriented. However, you can create structures and interfaces and that gives you 3 of 4 OOP principles: data abstraction, encapsulation, and polymorphism. Inheritance is the only one missing,
• Goroutines! — the greatest implementation of the light threads I have been ever using. It allows you to create a new thread in the super easy way using go operator and communicate between different goroutines using channels,
• It compiles to the single binary with all dependencies — no more packages’ conflicts!

Personally, I consider Golang as the greatest language which I use on the daily basis. However, this article won’t be about creating your first function or printing “Hello World”. I will show You a little bit more advanced stuff. If you are a beginner and want to learn more about Golang please visit its main page.

AWS Lambda & Golang

AWS Lambda is one of the most popular serverless compute services in the public cloud, released in November 2014 by Amazon Web Services. It allows you to run your code in response to the events like DynamoDB, SNS or HTTP triggers without provisioning or managing servers! Do you know what is really great? Since January 2018 it supports Golang runtime. Working with AWS Lambda is really simple - just upload a zipped package with your code and all the dependencies (single binary when using Golang).

Fast forward,4 years later at 2018 re:Invent AWS releases Lambda Layers that allows you to store and manage data that is shared across different functions in the single or even multiple AWS accounts! For example, While using Python you can put all the dependencies in an additional layer which can be later used by other Lambdas. There is no need to put different dependencies in each zipped packages anymore! In the Golang world situation is different as AWS Lambda requires you to upload compiled binary. How can we benefit from the AWS Lambda Layers? The answer is simple - build a modular application using Golang Plugins!

Golang Plugins — a way to build a modular application

Golang Plugins is the feature released in the Go1.8 that allows you to dynamically load shared libraries (.so files). It gives you an opportunity to export some of your code to the separate library or use the plugin prepared and compiled by someone else. It is promising, however, there are a few limitations:

• Your Plugin has to be single main module,
• You can load only functions and variables that are exported as ELF symbols,
• Due to static typing, you have to cast every loaded symbol to the correct type. In the worst scenario, you need to define the correct interface in your code,
• It works only for the Linux and MacOS. Personally, I do not consider this as a disadvantage :)

Building and testing your first plugin

Now let’s create our first plugin. As an example, we will create a simple module for the string encryption. Let’s go back to the basics and implement 2 simple encryption algorithms - Ceasar and Verman.

• Caesar cipher is the algorithm firstly used by Julius Ceases. It shifts every letter in the text by the fixed number of positions. For example, if you want to encrypt the word golang with the key 4 you will get ktpek. The decryption works in the same way. You just need to shift the letters in the opposite direction.
• Verman cipher is similar to the Ceaser, based on the same shifting idea, the difference is that you shift every letter by the different number of positions. To decrypt the text you need to have the key containing the positions used to encrypt the text. For example, if you want to encrypt the word golang with the key [-1, 4, 7, 20, 4, -2]you will get future.

The full implementation of this example is available here.

Plugin implementation

The following snippet contains the implementation of the two algorithms mentioned above. For each one we implement 2 methods of encrypting and decrypting our text:

As you can see, we exported here 3 different symbols (Golang exports only these identifiers that starts with the upper letter):

• EncryptCeasar - func (int, string) string that encrypts text using Ceasar algorithm,
• DecryptCeaser - func (int, string) string that decrypts text using Caeser algorithm,
• VermanCipher - variable of type vermanCipher implementing 2 methods: Encrypt: func (string) string and Decrypt: func () (*string, error)

To compile this plugin you have to run following command:

go build -buildmode=plugin -o plugin/cipher.so plugin/cipher.go

For now, there is nothing special - few simple functions were created and a module was compiled as a plugin by adding the -buildmode=plugin argument.

Load and test plugin

The fun begins when we want to use compiled plugin in our app. Let’s create a simple example:

First, you need to import the golang plugin package. It contains only two functions - the first one is for loading shared library and the second one is for finding an exported symbol. To load your library you have to use Open function which requires providing the path to your shared plugin and returns variable of type Plugin. If loading the library is not possible (ex. incorrect path or corrupted file) this function returns the error that has to be handled.

The next step is to load every exported symbol using the Lookup method. A slight inconvenience is that you need to load every exported function separately. However, you can combine multiple functions together in the same way as it was done for the VermanCipher symbol. Once you have loaded all the symbols you want to use you have to cast them to the correct type. Golang is statically typed language so there is no other way to use these symbols without casting. Remember, when you export a variable that implements a few methods, you need to cast it to the correct interface type (I had to define encryptionEngine interface to handle this).\newline \newline

To compile and run the app use the following command:

go build app.go
./app

In the output, you should see the encrypted and decrypted text as a proof that algorithm works correctly.

Use plugin in AWS lambda

To use our plugin in the AWS Lambda we need to make a few modifications in our application:

• AWS Lambda mounts layers to the /opt directory in the lambda container, so we have to load our plugin from this directory.
• We need to create a handler function that will be used by the Lambda engine to handle our test event.

The following snippet contains our application adjusted to be used by the Lambda:

As you can see the implementation is very similar to the previous one. We have only changed the directory from which we loaded our plugin and added the function response instead of printing values. If you want to learn more about writing Lambdas in golang please check the AWS documentation.

AWS Lambda deployment

There are two ways of deploying AWS Lambda functions and layers. You can create and upload zipped package manually or use the more advanced framework, which makes it much easier and faster. For most of my projects, I use the Serverless framework, so I have already prepared the simple serverless.yml configuration file using this tool:

service: cipherService
frameworkVersion: ">=1.28.0 <2.0.0"
provider:
  name: aws
  runtime: go1.x
layers:
  cipherLayer:
    path: bin/plugin
    compatibleRuntimes:
      - go1.x
functions:
  engine:
    handler: bin/cipherEngine
    package:
      exclude:
        - ./**
      include:
        - ./bin/cipherEngine
    layers:
      - {Ref: CipherLayerLambdaLayer }

In the layers section we defined a single layer with the path to the already created plugin - it will be deployed together with the lambda function. You can define up to 5 different layers which order is really important. They are mounted to the same /opt directory, so layers with the higher number can override files from the previously mounted layers. For every layer, you need to provide at least 2 parameters: path to the directory containing layer source (path to the plugin binary in your case) and the list of compatible runtimes.

The next functions section is a place where you define the list of the functions to be deployed. For every function, you need to provide at least the path to the compiled application. In addition, to that, we need to define the layers parameter with the reference to the layer defined above. This will automatically attach the layer to our Lambda function during the deployment. The funny thing is that you have to convert your lambda layer name to be TitleCased and add the LambdaLayer suffix if you want to refer to that resource. It seems that the Serverless team implemented it in this way to solve the conflict with reference to the different type of resources.

Once our serverless.yml configuration file is ready, the last thing to do is to compile our app, plugin and deploy it. We can use simple Makefile for that:

.PHONY: build buildPlugin clean deploy
build:
 dep ensure -v
 env GOOS=linux go build -ldflags="-s -w" -o bin/cipherEngine cipherEngine/main.go
buildPlugin:
 env GOOS=linux go build -ldflags="-s -w" -buildmode=plugin -o bin/plugin/cipher.so ../plugin/cipher.go
clean:
 rm -rf ./bin ./vendor Gopkg.lock
deploy: clean buildPlugin build
 sls deploy --verbose

You can build and deploy your function by running the following command:

make deploy

Test AWS Lambda

As I mentioned earlier AWS Lambda executes code in the response to the event. However we did not configure any event triggers, so it won’t be invoked without our help. We have to do it manually using the Serverless framework or the awscli tool:

sls invoke -f function_name
aws lambda invoke — function-name function_name output_file

In the response, you should see the same output as before, which proves that our lambda function works correctly and loads the plugin from the additional layer. Now you can create other functions that will use the same layer or even share it with other AWS accounts.

Summary

It was a lot of fun to use Golang modules and test how to integrate them with the newly released AWS Lambda Layers. Plugin library is really awesome, however because of its limitations and Golang specification it can be used only in some special scenarios. I think that for most of the developers who are working on the standard projects it won’t be needed or even possible to use plugins. Only two reasons come to my mind:

• Implementing complicated algorithms that can be used by the other applications ex. video coding or encryption algorithms.
• Sharing your algorithm with others without publishing its code.