Serverless webhooks — Designing Lift

This article is part of a series on Lift, an open-source project that simplifies deploying serverless applications.

As introduced in previous articles, Lift is a Serverless Framework plugin that simplifies deploying serverless applications.

At some point, every application needs to interact with an ecosystem of 3rd party SaaS providers. Implementing a webhook HTTP endpoint in your application allows this ecosystem of external applications to notify you. Your application can then react to those notifications and perform tasks accordingly.

As we approach the first beta planned in the next weeks, let’s deep dive about Lift’s 4th component, webhooks:

• How to deploy production-ready HTTP webhook endpoints.
• How we plan to simplify that with Lift.

The naive approach

Simple webhook endpoint architecture

API Gateway is the go-to solution to easily expose HTTP endpoints in your Serverless application. API Gateway is an AWS managed service handling authorization, routing and input validation.

You can natively integrate Lambda with API Gateway. Doing so, your code will be triggered every time an HTTP request hits a specific path of your application.

To deploy a webhook endpoint with the Serverless Framework, it’s as easy as a few lines of configuration:

# serverless.yml
# ...
functions:
  webhook:
    handler: webhook.handler
    events:
      # Bind our function to /webhook path for POST requests
      - httpApi: 'POST /webhook'

The missing characteristics of a good webhook design

The naive approach, while working, lacks some important criterion to maximize its efficiency. 3rd party apps notifiers expect your webhook endpoints to be available, respond fast and acknowledge all notifications. In order to do so, you should keep in mind:

• Transport / Processing separation of concerns: Webhooks are an inter-system notification medium based on HTTP transport. Acknowledging messages (authenticating, validating and parsing) and processing them are two different tasks: they should be done separately. As long as the message is acknowledged, our API should always return a 200 response. The response code should not depend on the ability of the application to use the notification content to perform some actions.
• Asynchronous decoupling: Webhooks are by nature an asynchronous notification protocol. A good webhook design should not put excessive stress on your application when the quantity of notification increases. A decoupling service (like a queue or a bus) should be implemented between the acknowledging and the processing workloads.
• Cost optimized: Webhook endpoints absorbs a various range of notification types. Your application might not want to actually do something for all of those types. It is therefore a good practice to ensure all of your infrastructure is not invoked for all incoming webhooks. Doing so, you’ll reduce risk of Denial of Wallet and be able to implement easily application filtering to process only the notification types you want.
• Error management: Processing error should not bubble up back to the notifier thanks to Transport/Processing separation of concern. However, some mechanism should be in place to ensure unprocessable message are not lost forever.
• Monitoring: You should be notified of excessive amount of notification you cannot authorize. This could either be a malicious attack on your endpoint or a wrong configuration of the secret used to check notification signature.

A production-ready approach

Production-ready serverless webhook endpoint architecture

Here is a preview of a minimal serverless.yml configuration that includes those best practices:

# serverless.yml
# ...
provider:
  eventBridge:
    useCloudFormation: true
# You can use native Serverless framework functionalities to define your processors
functions:
  typeAProcessor:
    handler: processores/typeA.main
    events:
      - eventBridge:
          eventBus: !GetAtt EventBus.Name
          pattern:
            source:
              - Webhook
            detail-type:
              - typeA
  typeBProcessor:
    handler: processores/typeA.main
    events:
      - eventBridge:
          eventBus: !GetAtt EventBus.Name
          pattern:
            source:
              - Webhook
            detail-type:
              - typeB
# But you need raw Cloudformation to define API Gateway integration to EventBridge
resources:
  Resources:
    # The event bus to publish webhook event to
    EventBus:
      Type: 'AWS::Events::EventBus'
      Properties:
        Name: WebhookBus
    # The HTTP Api used specifically for webhook
    ApiGateway:
      Type: 'AWS::ApiGatewayV2::Api'
      Properties:
        Name: WebhookApi
        ProtocolType: HTTP
    # Each API requires at least the $default stage
    ApiGatewayStage:
      Type: 'AWS::ApiGatewayV2::Stage'
      Properties:
        ApiId:
          Ref: ApiGateway
        StageName: $default
        # With auto-deploy, routes and integration are automatically available publicly
        AutoDeploy: true
    # Api Gateway needs a specific IAM role to be able to publish to EventBridge
    ApiGatewayRole:
      Type: 'AWS::IAM::Role'
      Properties:
        AssumeRolePolicyDocument:
          Statement:
            - Action: 'sts:AssumeRole'
              Effect: Allow
              Principal:
                Service: apigateway.amazonaws.com
          Version: '2012-10-17'
        Policies:
          - PolicyDocument:
              Statement:
                - Action: 'events:PutEvents'
                  Effect: Allow
                  Resource:
                    'Fn::GetAtt':
                      - EventBus
                      - Arn
              Version: '2012-10-17'
            PolicyName: EventBridge
    # The integration defines parameter mapping between HTTP request  body and EventBridge event properties
    ApiGatewayIntegration:
      Type: 'AWS::ApiGatewayV2::Integration'
      Properties:
        ApiId:
          Ref: ApiGateway
        IntegrationType: AWS_PROXY
        ConnectionType: INTERNET
        CredentialsArn:
          'Fn::GetAtt':
            - ApiGatewayRole
            - Arn
        IntegrationSubtype: EventBridge-PutEvents
        PayloadFormatVersion: '1.0'
        RequestParameters:
          DetailType: $request.body.type
          Detail: $request.body.payload
          Source: Webhook
          EventBusName:
            Ref: EventBus
    # Finally, the integration is deployed on a specific /webhook path of the Api
    ApiGatewayEndpoint:
      Type: 'AWS::ApiGatewayV2::Route'
      Properties:
        ApiId:
          Ref: ApiGateway
        RouteKey: POST /webhook
        Target:
          'Fn::Join':
            - /
            - - integrations
              - Ref: ApiGatewayIntegration

If we wanted to add the alarm, the dashboard as well as the lambda handling security check for the API Gateway endpoint, you’d need at least 100 additional lines of Cloudformation.

Needless to say:

• this is hard to figure out,
• this is tedious to implement.

We want to make that simpler with Lift.

Serverless webhooks with Lift

We are currently working on a “Webhooks” component that can be deployed via serverless.yml:

# serverless.yml
webhooks:
  stripe:
     authorizer: # accepts all function options
       handler: stripe/authorizer.main
     path: /my-webhook-endpoint # required : path the endpoint should be exposed at
    type: $request.body.type # optional : any static string or dynamically resolvable by API Gateway. Used to hydrate Eventbridge event's DetailType property

As we can see in the example above, there is a new webhooks section that lets us define webhooks (ideally, secluded by SaaS provider). In that instance, we define a stripe webhook.

On serverless deploy , Lift will create:

• An API Gateway v2 HTTP API with a route corresponding to the configured /my-webhook-endpointpath
• A stripe-authorizerLambda function, that will be invoked by API Gateway to ensure the notification signature is correct. This ensures the notification has not been tampered with and has been issued by Stripe.
• A WebhookBus Eventbridge bus that dispatch all notifications received from Stripe with an event containing the following properties:

{
  "Source": "stripe", // the webhook component name
  "DetailType": "payment_intent.succeeded", // body's type value
  "Detail": {
    // the whole notification body
  }
}

If not provided, the type option defaults to Webhook .

With the component deployed, you only need to register your processing Lambdas to perform specific operations:

# serverless.yml
# ...
provider:
  eventBridge:
    useCloudFormation: true
# You can use native Serverless framework functionalities to define your processors
functions:
  sendThankYouEmail:
    handler: sendEmail.main
    events:
      - eventBridge:
          eventBus: ${webhook:busName}
          pattern:
            source:
              - stripe
            detail-type:
              - invoice.paid
 
webhooks:
  stripe:
    authorizer:
      handler: stripe/authorizer.main
    path: /stripe
    type: $request.body.type

We’re working on a way to ensure all consumers are also configured with a DLQ and a retry policy. This will ensure no notification are lost in the process.

Deploying webhooks, queues, static websites and file storage is a small part of what we are working on with Lift.

To follow Lift’s development, star or watch Lift on GitHub.

We are also looking for feedback on the Webhook feature: get involved in the GitHub discussion.