3 ways to develop locally with webhooks

Webhooks are a lightweight, open, and effective way to work with events generated by other systems, and most importantly (for me at least) they avoid the need to use polling.

Despite their many benefits, one drawback that I’ve encountered in working with webhooks is their inability to be routed directly to your local development machine.

In this article I discuss 3 ways that I’ve found useful in working with that constraint.

A typical webhook set up

Just to clarify the issue a little further, when setting up a webhook on a 3rd party system you typically provide information on:

1. The “event” that you want to receive, e.g. new order created
2. The data to be sent, e.g. Order Id, Delivery Address, Order Amount etc.
3. Where you want the data sent to (the HTTP endpoint destination)

It is this last point where you can become a little unstuck as a developer wanting to work with these events on your local machine — what destination do you configure in order to get events delivered to localhost?

While there isn’t really a webhook standard, for the most part webhooks are typically implemented as HTTP POST requests with a JSON body payload sent to the consuming system (in this case your local machine).

As an example, if I attempt to configure a webhook on Marketplacer to send Invoice events to a localhost endpoint, I’d get the following response:

Error configuring a localhost webhook endpoint

And while you may get a slightly different error on other platforms, the fundamental issue remains, the 3rd party platform cannot route webhook events to localhost…

The remainder of this article discusses how you can work within this (understandable) constraint.

1. Fake it

This one is a bit of a cheat actually, and you’ll probably groan when I describe it, but in all honesty it’s an approach I’ve used, and indeed continue to use…

To use this approach all you have to do is use a HTTP API client like Postman or Insomnia and simply make direct HTTP POST requests to your localhost development endpoint, essentially “faking” a real webhook request. (Clients like Postman and Insomnia running locally on your development machine can make requests to localhost destinations).

In order to make this “fake” request as realistic as possible, you’ll need to ensure that you replicate it as close to the original webhook from the source system as you can. To do that you will need to:

• Generate a real webhook request
• Receive that webhook request somewhere (I use webhook.site for this)
• Examine the body payload and headers
• Replicate that body and headers with your HTTP Client (e.g. Postman)

If you navigate to webhook.site you’ll be given a unique, routable HTTP POST endpoint that you can use when setting up your webhook. Then when you trigger a webhook, this will be received at your webhook.site endpoint and you can examine, and replicate this in Postman.

Webhook captured by webhook.site

While this method may be considered a bit of a “cheat”, I continue to use it when I’m in the early stages of development, even if I have the other methods (listed below) available to me. Some of the reasons for this are:

• You don’t actually have to trigger a webhook on the source system which can sometimes be time consuming.
• You can easily manipulate the headers and body payload directly, again without having to generate an actual webhook event…

While this option is useful in the early (and possibly later) stages of development, it doesn’t provide you with a true end to end integration experience. In that case, you’d need to look at 1 of the other options…

2. Use an ingress app

The 2nd method requires the use of an ingress, or tunnelling app, with the most well known of these being Ngrok.

Ngrok (and similar apps such as Localtunnel), provide you with a routable url that maps down to a localhost endpoint on your dev machine. This means that you can provide the routable url to the system generating webhooks, and those webhooks will eventually make their way down to your local machine, and more specifically the app you’re building.

Simplified Ngrok architecture

The advantages of the this solution are:

• Very easy to set up (Ngrok & Localtunnel are anyway)
• Ngrok provides a decent free plan
• You get “real” webhooks delivered to your app

One of the potential downsides to this approach is of course the security considerations you may have in opening up a tunnel from your local machine to what is essentially a public endpoint. Depending on the industry you work in, or the organisation that you work for, using something like Ngrok may not even be an option available to you given the misgivings about potential security threats.

Setting up Ngrok

While the Ngrok documentation is clear and straightforward, I thought it may be useful to provide some steps for you here in case you’re interested in checking it out.

1. Sign up for an Ngrok account and download the app.
2. [Optional] Claim your free persistent endpoint. This means that you can use the same endpoint url every time you use Ngrok (otherwise you’ll get a newly generated endpoint url when Ngrok starts…)
3. Configure your authentication token. This token is provided to you under your Ngrok account settings along with the command prompt that you need to issue in order to add it to the Ngrok config file:

ngrok config add-authtoken <your auth token>

After that you can start Ngrok by specifying the port number of your local http endpoint, e.g.:

ngrok http 8080

This will start Ngrok with a non-persistent routable url, forwarding to: http://localhost:8080

Ngrok routing to http://localhost:8080

This did not suit my use-case as I wanted to:

• Use my persistent routable url (see step 2 above)
• Forward requests to a local https endpoint on port 8081 (so not just http)

In order to do this, you can alter the parameters you supply to the Ngrok cli in the following way:

ngrok http --domain=<persistent-domain> https://localhost:8081

In this case webhooks can be sent to https://<persistent-domain> and will be routed to my local https endpoint on port 8081

Ngrok can be configured way beyond this, but for the most part this covers what I need. However, as previously mentioned, ingress apps like Ngrok and Localtunnel may not be an option open to you - in which case there’s another option that could be adopted…

3. Use a proxy

There are a few variations of this pattern, but it essentially relies upon you having access to a routable endpoint somewhere (the “proxy”) which receives (and stores) the webhook traffic from the source system. You then need to retrieve the stored webhook messages down to your local machine, and finally route them to your local app.

As you can imagine there are a number of ways that this could be achieved but I use the following architecture:

Using Azure Functions and Service Bus to route webhooks locally

This approach relies upon 3 components:

1. An Azure HTTP Function
2. An Azure Service Bus
3. Service Bus subscriber

I have placed an example project on GitHub that you can use to get going with this architecture.

We talk a bit more about these below.

As you can see I happen to have a preference for Azure, but other public clouds will have comparable services that you could use if you wanted to implement this pattern.

Azure HTTP Function

This acts as a routable HTTP endpoint that can be used to directly receive webhooks from the source system. In this case I’ve elected to “store” these webhook events on an Azure Service bus. So when the Azure Function receives a webhook event it places it immediately on to the Service Bus.

Azure Service Bus

The Azure Service Bus acts as both:

• The persistent (or semi-persistent) store for the webhook messages
• The method of delivering those messages to a (local) subscriber

In relation to the persistence of the messages, the Service Bus does provide this but only for a relatively short period of time (when I say short, I’m talking about a period of days, or possibly weeks). This is more than enough for our use case however.

Service Bus Subscriber

This can be any app that can subscribe to our Azure Service Bus and read the messages from it. In this particular case this app is (not surprisingly) a .NET console app that triggers every time a message is placed onto the relevant topic of the Service bus. This agent then forwards these messages to whatever localhost endpoint it’s been configured to.

This approach may seem like using a sledge-hammer to crack a nut, but it does work and has the following advantages:

• Corresponds to more closely to something you’re likely to see in an enterprise production environment
• The use of a Service Bus brings with it all the advantages including, but not limited to: security, message routing, message persistence & event driven delivery.

The downsides of this approach are:

• Complexity. There are a few moving parts which makes the whole thing harder to configure, monitor and fault find than using something like Ngrok
• Cost. While the actual cost of this solution with a few hundred or even a few thousand webhooks would be minimal, it does still come with cost attached to it.

I did genuinely use this method recently when I was unable to use Ngrok (due to security concerns) so I know it works. However, for sheer ease of use, I’d always opt for the ingress option.