Webhooks Using AWS Lambda and API Gateway

How to handle real-time updates from 3rd parties via webhooks with highly scalable serverless AWS Lambda functions behind API Gateway

by the Whitespectre Node.JS Team

This article was co-authored by Benjamin Calace and Guillermo Caserotto, members of the Whitespectre Node.JS team.

We recently worked on a feature where we needed to integrate a third party solution into the application in order to process physical products asynchronously. In order to enable this, the third party solution provided webhooks, a technology that allows you to augment or alter the behavior of a web application. To do this, we used AWS Lambda and API gateways to handle webhooks in a serverless environment. In this article, we’ll take you through how this works and how you can set up a simple project to see it in action.

What you’ll find:

• Background on Webhooks
• Implementing a Serverless Webhook
• Setting Up a Simple Project

Background on Webhooks

First, some background on webhooks. The webhooks concept was coined by Jeff Lindsay in an article written back in 2007:

“The idea here is to create new infrastructure. New opportunities. I’ve been thinking a lot about the possibilities of a web hook enabled web, and it makes me really excited. Because it’s such open ended infrastructure, I’m sure the possibilities extend well beyond what I can think of. Especially when combined with our growing ecosystem of APIs and feeds.

Webhooks are easy to implement for application developers. You just implement a UI or API to let the user specify target URLs, and then you’re just making a standard HTTP POST on events. This is a fairly trivial operation in most environments, since you already do this to use other web APIs”.

A couple of years later webhooks were all over the Internet, enabling new infrastructure and extending functionalities of existing applications.

A webhook is basically a third party service which on some predefined event occurrence will, for example, trigger a POST on a predefined endpoint. The main benefit is that from our end, as webhook consumers, we wouldn’t need to constantly send requests to a regular API to see if there’s been an update or event occurrence, but rather we get notified of it as soon as it happens. This ensures that both our logic and their data is connected in almost real-time.

The webhook’s event can be anything, for example, the confirmation of a sale on an ecommerce site. The important thing is that this webhook does this automatically every time, so we only care about logic that can be done on our side, like for example, storing the IDs of the sold products.

In order to be able to use webhooks, the consumer will need to provide the webhook provider (the third party service) with an endpoint: any url which is publicly accessible will be enough for the provider to send HTTP requests to it. Ideally, both the provider and consumer need to agree on a way to ensure that the requests are indeed coming from the provider, and not some other possibly malicious user. A good way to do this would be sharing a secret key, which both the provider and consumer can hash their payloads with, and compare the resulting hashes.

Basically, if the third party’s webhook is set up, and we subscribed to it (they’d need to handle the appropriate event handling), all we need to work on is processing the webhook response.

Implementing a Serverless Webhook

One of the most efficient ways to handle webhooks is by using a serverless approach. And we can do this with an API gateway and Lambda from Amazon Serverless solutions.

An easy way to implement a Lambda/API Gateway is using AWS Serverless Application Model (AWS SAM) which is an open-source framework that allows you to define the application and then transform it to a CloudFormation project from scratch.

In the most simple way possible using an AWS Lambda function to process the webhook response, it could look like this:

Setting Up a Simple Project

Here’s an example using AWS SAM, AWS Lambda and an already existing provider could be as follows. Note: You need to have SAM CLI installed.

We can easily create a SAM project using the SAM CLI:

# sam init

The CLI will guide us through some questions in order to create our first project (note that values chosen are on each image):

Our project structure should look like this:

Inside app.js:

export async function lambdaHandler(event, context) {
 return {
   statusCode: 200,
   body: JSON.stringify({ message: 'hello from lambda' })
 };
}

A small sample configuration of template.yaml, which includes a Lambda exposed via API Gateway could be:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
 Simple Lambda and API Gateway
Resources:
 ApiGatewayEndpoint:
   Type: 'AWS::Serverless::Api'
   Properties:
     StageName: dev
 HandlerFunction:
   Type: AWS::Serverless::Function
   Properties:
     CodeUri: lambda-world/
     Handler: app.lambdaHandler
     Runtime: nodejs16.x
     Architectures:
       - x86_64
     Events:
       Handle:
         Type: Api
         Properties:
           RestApiId:
             Ref: ApiGatewayEndpoint   
           Path: /lambda-handler
           Method: post
     Environment:
       Variables:
         region: !Ref AWS::Region
         environment: dev
 
Outputs:
 ApiGateway:
   Description: "API Gateway endpoint URL for a sample Lambda function"

Then, if our AWS credentials are set up correctly, running sam deploy –guided would create an endpoint (API Gateway) which we’ll need to share with the webhook provider. The URL would run the lambdaHandler function, with the first param being the request payload and headers. We can of course test this function separately, either locally using sam local start-api or with our deployed endpoint. A good way to ensure everything ran smoothly is to write logs into CloudWatch (note: in JavaScript, a simple console.log() inside the Lambda would be written into CloudWatch if set up correctly).

Lastly, we need to make sure that our third party’s webhook knows of our new endpoint. This obviously would vary depending on the provider, but for example, a subscription to this webhook could be something like:

POST https://mycoolapp.com/webhook
 
{
   "someEvent": "item_sold",
   "url":"https://<<our-aws-url>>/dev/lambda-handler/"
}

Here, we assume that https://mycoolapp.com/webhook will fire an event (a POST request), whenever the “item_sold” occurs, pointed to the “url” sent during subscription, our API Gateway -> Lambda set. Inside lambdaHandler, event should contain all necessary metadata + payload data, relevant to this event.

So, that’s a very small example on how to listen on an existing webhook in a practical way, executing some logic inside a Lambda function. While this is just a sort of template, in theory we could add as much logic as we want inside the function, and listen to as many webhooks as we want.

Wrapping Up

So that’s a very small example on how to listen on an existing webhook in a practical way, executing some logic inside a Lambda function. While this is just a sort of template, in theory we could add as much logic as we want inside the function, and listen to as many webhooks as we want.

Overall, we’ve found that it’s quite straightforward to use an existing webhook with AWS, and extremely convenient to do it this way, due to ease of testing and deployment within AWS with SAM. It’s been really helpful in a lot of use cases where we need to connect two or more applications and execute the functionality we need when any event occurs.

Don’t forget to 👏🏻 if you enjoy reading our articles!

And subscribe to be the first to know what we are talking about 💡