Everyone has a webhook nowadays; whether it be monitoring a build on GitHub, receiving an analytics callback or ever processing customer payments… webhooks are becoming a dominant form of standardized communication between machines.
A common mistake developers make when implementing a webhooks receiver is doing too much within the receiving request itself.
Imagine you have a service that needs to recieve anywhere from 0–40,000 webhooks per minute. That’s a large variance that your server needs to be able to handle. If processing these webhooks is mission critical your service can’t fail under high load.
The solution to this problem is to use a decoupled producer (webhooks receiver) & consumer (job processor) connected via a job queue.
Recently, I had to implement this…
Your receiver & your processor should be two different things!
The most important thing about building a highly efficient and scalable receiver is to make sure it does as little as possible.
Ideally, all the endpoint does is receive a payload, enqueue the payload and return a 200 success method to the sender.
As soon as you think about doing any processing on the message upon receipt, you begin to add a potential failure point to your service. You can avoid processing in request by offloading the processing to a separate job, which reads the payloads from a queue.
Queueing services are plentyful… Some commercial/SAAS and some open source. I’m partial to Azure, and Azure storage queues are a super efficient solution.
While many people might see “Azure” and run away, I promise, it’s really awesome.
Azure Storage Queues
One of the great things about Azure storage queues, if you’re in the Azure ecosystem, they can easily function as a trigger for a web job. This is my preferred processing solution. You can learn more about the capabilities here. [Very specific Azure storage queue tip]
A common misconception is that these services are specific to Azure, but they work across many platforms & languages.
The pricing for the transactions is $0.0036 per 100,000. So before you even get to a cup of coffee, you can process 50 million requests (100 million transactions = $3.6. You’re going to use 2 transactions per webhook received).
Processing Your Data
I’m not going to dive into this, since there are a myriad ways to go about this. I choose Azure webjobs because they integrate so well with the storage queues. But really, you can run it in real time or batches, however you choose. The key is that it runs separated from your web hook ingestion, and sudden load increases won’t choke the processing or bring down your service.
Results
The screenshot at the top of this post shows a sample from Azure insights from this endpoint. It had absolutely no issues handling tens of thousands of requests per minute with response times averaging 26 ms.
I hope this was helpful!
If you enjoy, and want to learn more, let me know on twitter or via my website.
Don’t forget to hit that 💚.
