Serverless webhooks — Designing Lift
This article is part of a series on Lift, an open-source project that simplifies deploying serverless applications.
As introduced in previous articles, Lift is a Serverless Framework plugin that simplifies deploying serverless applications.
At some point, every application needs to interact with an ecosystem of 3rd party SaaS providers. Implementing a webhook HTTP endpoint in your application allows this ecosystem of external applications to notify you. Your application can then react to those notifications and perform tasks accordingly.
As we approach the first beta planned in the next weeks, let’s deep dive about Lift’s 4th component, webhooks:
- How to deploy production-ready HTTP webhook endpoints.
- How we plan to simplify that with Lift.
The naive approach
API Gateway is the go-to solution to easily expose HTTP endpoints in your Serverless application. API Gateway is an AWS managed service handling authorization, routing and input validation.
You can natively integrate Lambda with API Gateway. Doing so, your code will be triggered every time an HTTP request hits a specific path of your application.
To deploy a webhook endpoint with the Serverless Framework, it’s as easy as a few lines of configuration:
# serverless.yml
# ...functions:
webhook:
handler: webhook.handler
events:
# Bind our function to /webhook path for POST requests
- httpApi: 'POST /webhook'
The missing characteristics of a good webhook design
The naive approach, while working, lacks some important criterion to maximize its efficiency. 3rd party apps notifiers expect your webhook endpoints to be available, respond fast and acknowledge all notifications. In order to do so, you should keep in mind:
- Transport / Processing separation of concerns: Webhooks are an inter-system notification medium based on HTTP transport. Acknowledging messages (authenticating, validating and parsing) and processing them are two different tasks: they should be done separately. As long as the message is acknowledged, our API should always return a
200
response. The response code should not depend on the ability of the application to use the notification content to perform some actions. - Asynchronous decoupling: Webhooks are by nature an asynchronous notification protocol. A good webhook design should not put excessive stress on your application when the quantity of notification increases. A decoupling service (like a queue or a bus) should be implemented between the acknowledging and the processing workloads.
- Cost optimized: Webhook endpoints absorbs a various range of notification types. Your application might not want to actually do something for all of those types. It is therefore a good practice to ensure all of your infrastructure is not invoked for all incoming webhooks. Doing so, you’ll reduce risk of Denial of Wallet and be able to implement easily application filtering to process only the notification types you want.
- Error management: Processing error should not bubble up back to the notifier thanks to Transport/Processing separation of concern. However, some mechanism should be in place to ensure unprocessable message are not lost forever.
- Monitoring: You should be notified of excessive amount of notification you cannot authorize. This could either be a malicious attack on your endpoint or a wrong configuration of the secret used to check notification signature.
A production-ready approach
Here is a preview of a minimal serverless.yml
configuration that includes those best practices:
# serverless.yml
# ...provider:
eventBridge:
useCloudFormation: true# You can use native Serverless framework functionalities to define your processors
functions:
typeAProcessor:
handler: processores/typeA.main
events:
- eventBridge:
eventBus: !GetAtt EventBus.Name
pattern:
source:
- Webhook
detail-type:
- typeA
typeBProcessor:
handler: processores/typeA.main
events:
- eventBridge:
eventBus: !GetAtt EventBus.Name
pattern:
source:
- Webhook
detail-type:
- typeB# But you need raw Cloudformation to define API Gateway integration to EventBridge
resources:
Resources: # The event bus to publish webhook event to
EventBus:
Type: 'AWS::Events::EventBus'
Properties:
Name: WebhookBus # The HTTP Api used specifically for webhook
ApiGateway:
Type: 'AWS::ApiGatewayV2::Api'
Properties:
Name: WebhookApi
ProtocolType: HTTP # Each API requires at least the $default stage
ApiGatewayStage:
Type: 'AWS::ApiGatewayV2::Stage'
Properties:
ApiId:
Ref: ApiGateway
StageName: $default
# With auto-deploy, routes and integration are automatically available publicly
AutoDeploy: true # Api Gateway needs a specific IAM role to be able to publish to EventBridge
ApiGatewayRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: 'sts:AssumeRole'
Effect: Allow
Principal:
Service: apigateway.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action: 'events:PutEvents'
Effect: Allow
Resource:
'Fn::GetAtt':
- EventBus
- Arn
Version: '2012-10-17'
PolicyName: EventBridge # The integration defines parameter mapping between HTTP request body and EventBridge event properties
ApiGatewayIntegration:
Type: 'AWS::ApiGatewayV2::Integration'
Properties:
ApiId:
Ref: ApiGateway
IntegrationType: AWS_PROXY
ConnectionType: INTERNET
CredentialsArn:
'Fn::GetAtt':
- ApiGatewayRole
- Arn
IntegrationSubtype: EventBridge-PutEvents
PayloadFormatVersion: '1.0'
RequestParameters:
DetailType: $request.body.type
Detail: $request.body.payload
Source: Webhook
EventBusName:
Ref: EventBus # Finally, the integration is deployed on a specific /webhook path of the Api
ApiGatewayEndpoint:
Type: 'AWS::ApiGatewayV2::Route'
Properties:
ApiId:
Ref: ApiGateway
RouteKey: POST /webhook
Target:
'Fn::Join':
- /
- - integrations
- Ref: ApiGatewayIntegration
If we wanted to add the alarm, the dashboard as well as the lambda handling security check for the API Gateway endpoint, you’d need at least 100 additional lines of Cloudformation.
Needless to say:
- this is hard to figure out,
- this is tedious to implement.
We want to make that simpler with Lift.
Serverless webhooks with Lift
We are currently working on a “Webhooks” component that can be deployed via serverless.yml
:
# serverless.ymlwebhooks:
stripe:
authorizer: # accepts all function options
handler: stripe/authorizer.main
path: /my-webhook-endpoint # required : path the endpoint should be exposed at
type: $request.body.type # optional : any static string or dynamically resolvable by API Gateway. Used to hydrate Eventbridge event's DetailType property
As we can see in the example above, there is a new webhooks
section that lets us define webhooks (ideally, secluded by SaaS provider). In that instance, we define a stripe
webhook.
On serverless deploy
, Lift will create:
- An API Gateway v2 HTTP API with a route corresponding to the configured
/my-webhook-endpoint
path - A
stripe-authorizer
Lambda function, that will be invoked by API Gateway to ensure the notification signature is correct. This ensures the notification has not been tampered with and has been issued by Stripe. - A
WebhookBus
Eventbridge bus that dispatch all notifications received from Stripe with an event containing the following properties:
{
"Source": "stripe", // the webhook component name
"DetailType": "payment_intent.succeeded", // body's type value
"Detail": {
// the whole notification body
}
}
If not provided, the type
option defaults to Webhook
.
With the component deployed, you only need to register your processing Lambdas to perform specific operations:
# serverless.yml
# ...provider:
eventBridge:
useCloudFormation: true# You can use native Serverless framework functionalities to define your processors
functions:
sendThankYouEmail:
handler: sendEmail.main
events:
- eventBridge:
eventBus: ${webhook:busName}
pattern:
source:
- stripe
detail-type:
- invoice.paid
webhooks:
stripe:
authorizer:
handler: stripe/authorizer.main
path: /stripe
type: $request.body.type
We’re working on a way to ensure all consumers are also configured with a DLQ and a retry policy. This will ensure no notification are lost in the process.
Deploying webhooks, queues, static websites and file storage is a small part of what we are working on with Lift.
To follow Lift’s development, star or watch Lift on GitHub.
We are also looking for feedback on the Webhook feature: get involved in the GitHub discussion.