Webhook Processing with API Gateway and SQS

Background

The explosion of Software as a Service (SaaS) products over the last few years has meant that engineers need to find reliable ways to handle webhook invocations. Webhooks are generally used by SaaS products to notify clients of events, in real-time, as they occur. They normally take the form of HTTP POST requests with a JSON body.

Some SaaS products have some level of reliability handling built into their webhook offerings. For example, if the endpoint being invoked returns anything other than a 2xx response code, then the SaaS product might retry the request over the course of the next few hours.

While this level of error handling is better than nothing, minimizing the need to leverage external reliability systems is ideal. In other words, any processing performed on a webhook invocation must be as simple as possible, with the goal of reducing the risk of errors. Along these lines, it is preferable to divorce the logic of receiving the request and storing it safely from the business logic of processing the request.

Pattern for Webhook Processing

In order to keep processing as simple as possible, the following pattern has served me well in the past:

1. SaaS invokes webhook endpoint.
2. Endpoint authenticates the request and offloads the payload into a staging system. No synchronous business logic is performed.
3. 2xx response code is returned. If the request cannot be authenticated or there is an error getting the payload into a staging system, an error is returned.

The staging system identified in the second step could be a variety of things:

• A queuing systems (such as AWS Simple Queue Service)
• A streaming system (such as AWS Kinesis)
• A data store (such as AWS DynamoDB)

Regardless of the staging system, the point here is to minimize the moving parts (and therefore the likelihood of errors).

Webhook Processing with API gateway and SQS

Concrete Example: API Gateway and SQS

In this example…

1. We will create an SQS queue capable of storing webhook requests.
2. We will create an IAM role such that an API Gateway endpoint can talk to SQS.
3. We will create an API Gateway endpoint capable of receiving POST requests with a JSON body. On receiving a request, it will send the message to SQS.

Create an SQS Queue for Storing Webhook Requests

1. Log into AWS and go into the SQS landing page.
2. Click “Create queue”

Click “Create queue”

3. Create a Standard queue and give it a name. Other options can be left at their default values (exploring other queue options is out of scope for this article).

Name your new Standard Queue

Once you click “Create queue” your new queue will be available for use.

Successful queue creation

Create a Role for API Gateway to Access SQS

Before creating our API Gateway endpoint, we need to create a role that gives permission for API Gateway to access SQS. In our example, we will be very broad but quickly gets us to a working state. Production configurations should be as secure as possible.

1. Log into AWS and go into the Identity and Access Management (IAM) landing page.

2. Click “Roles” on the left pane and “Create role”

3. Select “AWS Service” as the trusted entity type.

4. Select the “API Gateway” use case and click “Next: Permissions”

5. Click “Next: Tags” and then click “Next: Review”

6. Give your new role a name and click “Create role”

7. Find your new role in the list and click it

8. Click “Attach policies”

9. Search for “sqs” and select “AmazonSQSFullAccess” and click “Attach policy”

Your new role now has permissions to access SQS.

10. Copy the Role ARN for later. You will need to specify it when we create the API Gateway endpoint that integrates with SQS.

Create an API Gateway Endpoint for Receiving Webhook Requests

1. Log into AWS and go into the API Gateway landing page.
2. Click “Create API”

Click “Create API”

3. Scroll down to “REST API” and click “Build”

4. Give your API a name and click “Create API”

Give your API a name and click “Create API”

5. Create a new resource. Name it “webhook”

6. Create a new POST method under the “webhook” resource

7. Configure it as follows:

• Integration Type: AWS Service
• AWS Region: Your region (in my case, I am using us-east-1)
• AWS Service: Simple Queue Service (SQS)
• AWS Subdomain: leave blank
• HTTP method: POST
• Action Type: Select “Use path override”
• Path override: Type the path to the queue, as:

AWS Account Number/queue name. Side note: I’m not sure why this is not simply the ARN.

Example: 1234567890/my-fun-queue

• Execution role: Enter the ARN for the role we created above.
• Content Handling: Passthrough
• Use Default Timeout: Checked

8. Add a new HTTP Header.

• Name: Content Type
• Mapped from: ‘application/x-www-form-urlencoded’. (quotes are important)

9. Add a new Mapping Template. Select “Never” for Request Body Passthrough. Add a new mapping template for “application/json”.

10. Specify the “application/json” template as follows and click “Save”

Action=SendMessage&MessageBody=$input.body

At this point, everything should be in place and we should be ready to test!

Testing it Out

Your new “webhook” endpoint should look like this:

1. Click the “Test” button.

2. Populate a request body (this can be anything — this simulates what the SaaS product might send in the JSON request body) and click “Test”

3. You should see a “200” response status that indicates it went as expected.

4. Navigate to SQS and your queue. Click “Send and receive messages”

5. Under “Receive messages” you should see one message available. Click “Poll for messages” to see the message payload.

6. Click on the message

7. Click “body” to view the payload

In this article, a simple pattern for handling webhook requests was identified and a concrete example provided. Using this pattern, you can divorce the logic of receiving and storing webhook requests from the actual business logic of processing the request.