Processing webhooks with Node.js

Ingest, transform and route HTTP requests with no server, and no cloud resources. Just Node.

At Pipedream, we process webhooks from every service that provides them. We usually write (Node) code to transform and act on these webhooks, but don’t want to run an HTTP server or serverless stack for each one.

We developed Pipedream in part to solve this problem, so I’d like to show you what Pipedream is and how you can use it to run Node code on any webhook or HTTP request.

First — what are webhooks, and what can you use them for?

If you’ve ever tried to get data out some app into your system, you’ve probably heard of webhooks.

Stripe, like many services, supports both webhooks and an API

Webhooks enable services like Stripe, Github, or Google to push data to you, instead of you having to pull data from them.

When some event occurs (e.g. someone buys something from your store), the service sends you an HTTP request with data about that event. You don’t have to run a script to poll API for new events; they’ll tell you when they happen, as soon as they happen.

Github for example, can send you a request each time someone opens a new issue in your repo. They’ll tell you when a new branch is created, or even when a repo is starred. Github calls these events, and that’s how we’ll refer to them below.

If a service supports webhooks, you can automate almost anything where you need to take some action in response to some event.

For example, you can send yourself an email or Slack message when an event occurs in your service, or trigger an action in another service (e.g. update Salesforce each time you get a Stripe subscription).

If you want to analyze data from a service, you can send webhooks directly to a database or data warehouse as a record of every event. This lets you build charts or run SQL queries to ask questions you can’t answer using the service’s built-in dashboards. You can also join these events with data from your own or other apps.

There’s a lot you can do with webhooks. Now you’ve just got to write the code to do it!

What tech can you use to process webhooks?

Webhooks are just HTTP requests. Any service that can process HTTP requests can process webhooks.

HTTP server running on a VM, or as a container

An example web app to process Github webhooks

Any HTTP server (e.g. nginx) can accept webhooks and route them to a web app (you might have used Express, or Flask, to build such an app). But setting up these services takes time. It’s easy to run an Express app on your local machine, but deploying it to an EC2 instance, or as a Kubernetes service, can be daunting for beginners. Even for experts, there’s a lot to maintain and monitor.

For certain applications, this stack is ideal. For processing webhooks, it’s often overkill.

The serverless option

An example of AWS services to process Github webhooks

Thankfully, better options exist. AWS and other cloud platforms provide “serverless” tools like API Gateway and Lambda that facilitate the creation of HTTP services without you having to run a server. Once you get the hang of them, these tools let you sing: you can spin up applications in minutes, and for a use case like this, the serverless options are likely to be cheaper than the EC2 equivalent.

AWS provides a network of powerful services you can mix and match in myriad ways. But since these services do one thing and one thing well, it requires at least 4 (API Gateway, Lambda, IAM, CloudWatch) to run a basic web app. And learning AWS is hard.

Again, if each part of this stack serves a clear purpose for you, this option is ideal. But for processing most webhooks, I’d argue it’s still too low-level.

Pipedream

Run Node code directly on Pipedream’s infra

My team and I are building Pipedream because we believe there’s a simpler way to run these services. When you build an Express app to process HTTP requests, it’s trivial to run it locally:

> node app.js

Deploying it to the public internet? Not so easy.

If a part of your stack doesn’t directly implement your business logic, it shouldn’t be a part of your stack. Since most of your logic is implemented in code, you should spend most of your time writing code.

We believe that you should just be able to write code that runs on HTTP requests. Someone else should worry about the HTTP server, app configuration, execution environment, and scale. This is our goal with Pipedream.

What is Pipedream?

Pipedream is a developer automation platform that lets you run any Node.js code or pre-built actions on HTTP requests, for free, without managing a server.

A quick overview of Pipedream. Click on the GIF to learn more.

Instead of cloud functions or web apps, you build workflows. A workflow is a linear sequence of steps (code, actions) triggered by some event, like an HTTP request.

When you create a workflow, you choose the Source for your events. Choosing the Webhook source will generate a URL specific to this workflow:

This URL is unique to your workflow. Any HTTP requests sent here will trigger the steps of the workflow.

Each time you send an HTTP request to this URL, we’ll run your workflow top-to-bottom. Within code and action steps, you can access properties of the HTTP request using the event object. For example, event.body contains the the payload (body), and event.headers has the headers. You can also share data between code and action steps.

console.log statements and other standard output is logged under the step where it was run

Pipedream runs your workflows for you. You manage no server and no cloud resources. You sign up with Github or Google, write code, and we run that code for every HTTP request you send.

Workflows are free to run up to 10 times per second. That’s roughly 25 million executions per month, per workflow, for free, and you can run as many workflows as you want.

Finally, workflow code is public (the events you send to workflows, and the logs they generate, are completely private). When you create a workflow, anyone with the link can see its code. They can fork and use that workflow on their own. Over time, we hope y’all will build workflows to solve your own problems, which others can fork and use in one click, saving them from having to build that same code from scratch.

Using Pipedream to process webhooks

We’ll show you how to solve two specific problems using Pipedream:

1. Send yourself an email each time someone stars your Github repo.
2. Send webhooks to S3, SQS, Kinesis, or any AWS service.

If you don’t use Github or AWS, or don’t need to solve these specific problems, you should still follow along with the examples. Just replace the service with your own webhooks; all the same concepts apply.

Problem #1: Github Star to Email

Let’s say you wanted to email yourself each time someone starred a repo you owned:

Github can send you webhooks on stars, and you can use Pipedream to ingest those webhooks and send that email in just a few minutes.

First, open this Pipedream workflow in a new tab, and click on the Copy button:

Forking a workflow creates a copy of it in your account that you can run and modify

You can run and modify this copy however you’d like. If you haven’t signed up for Pipedream, you’ll be asked to during the copying process.

As soon as you Copy, you should see a URL at the top of your workflow:

This URL is unique to your workflow. Any HTTP requests sent here will trigger your workflow.

This URL is unique to your workflow. Any HTTP requests sent here will trigger your workflow and run it on the specific HTTP request. Copy this URL.

In Github, go to your repo’s Settings tab, then the Webhooks option on the left. Click the button to Add webhook.

Paste the Pipedream webhook URL into the Payload URL field, set the Content type to application/json, and optionally, add a secret (we’ll show you how to validate this secret later).

Finally, under the Which events would you like to trigger this webhook? section, select the option to Let me select individual events. Uncheck Push, check Stars:

This will ensure you only get webhook notifications on stars.

Click the big green Add webhook button. This will trigger the Github ping event, which informs you that the webhook has been created successfully. Back in your Pipedream workflow, you’ll see a new event appear in the event inspector, on the left:

We’ve written this workflow to ignore the ping event, exiting early if we see it:

The $end function immediately ends a workflow

The $end function immediately ends a workflow. No code or steps below that point are run. $end accepts an optional string that, if present, appears in the event inspector so you can quickly see why your workflow ended early.

Now, go back to your Github repo and Star your repo:

In Pipedream, you’ll see another event appear in the inspector. Select it, and scroll to the bottom of the workflow, to the Send Me An Email step:

The email action lets you send an email to yourself without writing code. Notice that the Subject and Message fields reference properties of the event object, which contains the data sent from Github in event.body. Since we configured Github to send JSON payloads, Pipedream also converts that payload — event.body — to a JavaScript object, so you can easily reference its properties in code and actions steps.

For example, event.body.sender.login contains the username of the user who starred the repo, and event.body.repository.name contains the name of the repo. Each time the workflow is run on a new event, we extract the values of these properties and format the email accordingly.

Check your inbox. You should see an email that looks something like this:

Problem #1 Bonus: validate the Github webhook secret

Earlier, we added a Secret to our Github webhooks. Github takes that secret, and the HTTP payload it sends with each webhook, and computes a hash. It sends that hash in the X-Hub-Signature HTTP header with every request.

If Pipedream also knows the secret you entered in Github, we can compute the same hash and validate this request came from Github, instead of from another server pretending to be Github.

This Pipedream workflow contains the same code as above, with an additional code step to validate the Github signature.

Since Pipedream workflows are public, we want to make sure our Github secret isn’t included in our workflow code. Instead, we can create a Pipedream environment variable called GITHUB_SECRET, then reference that value as process.env.GITHUB_SECRET in our workflow. We use the value of our secret, and the HTTP payload, to compute and validate the signature:

Like our other code steps, this will run on every request from Github. If you’ve configured everything correctly, you shouldn’t see any output from this step — it’ll validate the signature and move on.

Problem #2 — Store webhooks in S3, SQS, etc.

At Pipedream, we frequently ingest data from third party services (like Sendgrid, Wistia, and Github) to run analysis on it, joining the events with our internal data or just getting deeper insight into the data than we can get through the service’s dashboards.

We store all webhooks in Amazon S3, and use Pipedream to do it. This Pipedream workflow accepts HTTP requests and stores the HTTP payloads to S3.

Every HTTP request sent to this URL will get sent to your S3 bucket

Once you Fork the workflow,

1. Add this bucket policy to your bucket, allowing Pipedream to send data to it
2. Enter your bucket name in the Bucket field of the Send to Amazon S3 action. Optionally, you can add a Prefix (folder) where we’ll store your data in your bucket.
3. Send an HTTP request to your URL. If you don’t have a webhook you’d like to send, you can use cURL or press the Send Test Event button to send a test HTTP request.

When you send data to S3, we handle a few things for you:

• We’ll collect the requests sent to the workflow and, once a minute, send them in a batch. If you send 100 requests within a 60 second period, those requests will all end up in a single S3 object.
• We gzip compress this object for you.
• We also write objects to the bucket using the following format:

[PREFIX]/YYYY/MM/DD/HH/YYYY-MM-DD-HH-MM-SS-IDENTIFIER.gz

for example, if you use a prefix of test/, you might see an object in your bucket at the following path:

s3://my-s3-bucket/test/2019/05/25/16/2019-05-25-16-14-58-8f25b54462bf6eeac3ee8bde512b6c59654c454356e808167a01c43ebe4ee919.gz

We believe this is one of the easiest ways to stream data to S3, but we’re looking for your feedback on we can simplify and improve it.

Want to send webhooks to SQS, Kinesis, or other services? You can use the AWS JavaScript SDK within any Pipedream workflow to interact with these services in any way. These example workflows store data to SQS and Kinesis, so you can use them as examples for working with other services, too:

• Webhook to SQS
• Accept HTTP requests, send payload to Kinesis

What else can Pipedream do?

There’s a lot we haven’t covered. You can:

• Use more of our pre-built actions to send events to Slack or Discord, store state in KVdb, and more.
• In addition to the Email and S3 destinations, you can send data to an HTTP endpoint (creating your own webhooks), Snowflake, and Server-sent Events (SSE) streams.
• You can send data to the Pipedream Data Warehouse and run SQL queries on it without managing your own data warehouse or database. You can create as many tables as you want, and store as much data as you want. We’ll retain the data you send here for 30 days.

There’s a lot more on our backlog, and we’re looking for guidance from devs on what to build. Are you running a web app that’s a pain to maintain that could it be run on Pipedream, but need feature X to do it? We love feedback and want to hear what would help y’all be more productive. Leave us a comment below or reach out to let us know!