Mapping Templates: Transforming Your Payloads Within AWS API Gateway

Introduction

When developing microservices, we often need to transform the raw data that gets passed as input to our APIs so that it can be interpreted by downstream services. It is definitely the case for some of our services and in this article I will present the case for leveraging AWS Gateway capabilities to do most of the heavy lifting for us.

Establishing the Base Context

To guide our discussion, let’s consider the following scenario:

Fig. 1 — Call to notification service with initial input

Here we have a Payment application that sends a POST request to the Notification service with a certain payload. The Notification service will then handle the request and eventually forward it to a third-party API service for further processing (i.e. external logging service). However, it can’t simply pass the payload as-is; it must adhere to the contract specified by the service provider. Therefore, a transformation of the original payload is needed in order to map the data to the expected contract.

Fig. 2 — Initial input transformed to 3rd-Party Messaging API format

Here we see that the original payload now takes on a new form where certain attributes are mapped to different properties. We even had to encode the entire payload as base64 and map its value to the data attribute.

In a typical client-server application where our backend is a simple Node Express microservice hosted on a Kubernetes cluster, for example, we could create an adapter that converts given input to the desired format before passing it to any downstream services (see Adapter pattern).

But what if our solution was serverless instead? In this case, our service can be composed of several different components, each performing distinct functions that can also inter-communicate with each other. One such approach is using AWS API Gateway to define an HTTP API interface that can invoke other AWS services (i.e. Lambdas, Step Functions and EventBridge) as well as integrate with other HTTP services.

Let’s take our previous Notification service and convert it to an AWS serverless application. To provide a more concrete example, I’ve replaced the 3rd-party Message API with AWS EventBridge service instead.

Fig. 3 — Initial Notification service as AWS Serverless solution

In this scenario, we now need to make 2 transformations before sending the data to EventBridge:

1) Map incoming payload to match downstream service contract

2) Map the resulting payload from the previous transformation to match EventBridge’s PutEvents contact

Using Lambdas

Our first reaction might be to add a Lambda in between to handle the transformation.

Fig. 4 — Using Lambda function as an Adapter between API Gateway and EventBridge

Although this is a viable solution it has a few drawbacks.

• Lambdas can be costly: Cold starts can significantly increase latency and affect overall performance. They can become a bottleneck, especially during high-traffic launches as we’ve discovered here at SSENSE during our load testing experiments.
• Reserved versus Provisioned concurrency for high traffic events:
  - Need to fine-tune accordingly to avoid overage costs
  - It is not always straightforward as you might want to do some load testing in order to figure out the optimal configuration
• Extra code/infrastructure to maintain

Is there a way to apply this type of transformation without having to use Lambdas? Yes!

Request Mapping Templates to the Rescue

The type of transformations we need can be achieved by using API Gateway’s Mapping Template functionality. This allows us to perform basic to complex transformations, like the one described earlier, without the need to add any additional resources to our stack.

Request mappings can be defined via the API Gateway console or as part of the OpenAPI definition that can be used to create or update an API. I will demonstrate how this can be done using the OpenAPI method with our previous example.

Here’s what the OpenAPI specification could look like for our API:

Notice that under the post object, there’s a x-amazon-apigateway-integration extension property. This allows us to define additional properties pertaining to API Gateway Integrations for that specific endpoint. One such property is requestTemplates which contains the mapping script for each MIME type we want to support.

In this example, I have a template for application/json MIME type written in Velocity Template Language (VTL):

In the first line of the script we set a variable eventData with the payload that we receive as input using JSONPath expression. The following line sets another variable encodedMsg that will store the result of our first mapping.

Notice the use of $util variable to perform certain transformations:

• escapeJavaScript transforms the input header value (x-request-id) into a Javascript-friendly string
• base64Encode transforms the original input (eventData) into a base64 string

This is provided by API Gateway and is just one of many variables and utility functions that you can use for your mapping script (see here for the complete list).

Once we’ve defined our first mapping, we then define our 2nd mapping (lines 13 to 22) which is the payload that will be sent to EventBridge. Notice that just before (L12) however, we set our first transformed payload (encodedMsg) in eventData’s encoded property. We do this to stringify the payload in JSON format using the $input.json function (L16).

And there you have it! With just 20 lines of code, we managed to apply 2 payload transformations without using any additional resources such as Lambda, all within API Gateway’s request template functionality. You can find other examples of this in AWS documentation here.

Although Mapping Templates are extremely powerful, they should be used with caution. We’ve seen that they can handle some transformations quite well, but in some cases we’ve noticed that they are not always applicable. One such case is when your API has to support non-text MIME types like multipart/form-data or application/x-www-form-urlencoded where you would need to parse data as a byte stream and/or use special libraries to perform additional parsing (e.g. CGI in Python). In this case, having a Lambda in between your API Gateway instance and consuming resources (EventBridge/Kinesis/Step Function) as an adapter is a more viable option.

Another caveat for using Mapping Templates is that it requires you to use VTL to script the mapping. Unless you’re very familiar with this language and its quirks and intricacies, it can be quite the learning curve at first (as I discovered myself).

Furthermore, there’s no official way to programatically unit test VTL scripts locally on your machine; you have to go by trial and error using the Test Method feature in the API Gateway console within a sandbox environment. Although there are some third-party modules that provide this functionality (like this), they’re usually limited to one language (Javascript) and/or are outdated, without any support from AWS. Perhaps AWS will eventually provide support for this as more and more developers request this feature.

Conclusion

One of the mantras of cloud computing is to offload, as much as possible, to the infrastructure. If you use API Gateway, consider using Mapping Templates. This way you can apply several types of transformations to your payload, from simple extraction to parsing and encoding the input data, without the need for additional resources like Lambdas which can become a performance bottleneck to your overall architecture. They provide great flexibility in performing common transformations such as many-to-one parameter mapping, encoding values prior to mapping, injecting new parameters and mapping parameters based on conditions.

This does not apply to every use case, particularly if your payloads aren’t in JSON format. For more complex data types (i.e. binary files), it’s best to go with a Lambda or some other computational resource. Although API Gateway provides some utility functions like base64 encoding, it doesn’t provide the parsing utilities that are usually required to deal with such data.

Happy templating!

Editorial reviews by Catherine Heim & Mario Bittencourt

Want to work with us? Click here to see all open positions at SSENSE!