Why I created Postbacks — the callback scheduler
The problem seemed simple. I just needed to send a postcard at the date & time a user requested. It could be tomorrow, it could be in 3 months. I figured there must be a service which would call me back, and tell me which postcard to send when it came time. To my surprise, nothing catered to this specific use case.
In comes Postbacks…
What I needed was a solution that would call me back at the right time, with the right data. A solution I could easily see the status of all my scheduled tasks, get alerts as they failed, and re-send tasks as necessary.
Simple right??
Sure, as long as making 100,000 concurrent HTTPS calls is easy.
I started with a basic assumption. At any minute, we need to be ready to execute 100,000+ HTTP requests. That’s 6 million tasks per hour, 144 million per day, and 4.5 billion per month.
Not only do we have to execute 100,000+ requests, we have to do it in a timely manner. People are waiting for their callbacks after all. I set the threshold to 15 seconds.
How do we do it?
Let's assume we have 1 billion postbacks queued to be run over the following year. At this current minute, our job is to process 100k of them within 15 seconds.
The first step is finding those 100k records. Searching through a database of a billion records would waste precious seconds. Instead, we utilize blob storage. Each postback is appended to a blob in accordance with the datetime it will run.
One second later…
We now have the 100k requests. How can we economically and efficiently send these callbacks in under 15 seconds? Enter Azure Functions. Azure Functions allow us to pay for only the resources we use, and scale as needed. If you’re not familiar with Azure Functions, you can read more here. Instead of paying for the biggest, badest server, with infinite processing power, and a plethora of threads, we can simply spin up Azure Functions as they’re needed.
500 requests per ~10 seconds
Utilizing async methods, testing showed that a single Azure Function could crank through about 500 requests in ~10 seconds. This meant anytime there were more than 500 requests to be processed per minute, we would need some sort of concurrency to ensure all requests were made in under 15 seconds. Here lies the power of Azure Functions.
Azure Functions can be serverless, meaning they’re essentially their own thread. This allows us to call an infinite (or very large) amount of azure functions to do our processing. In our case, we process the postbacks in batches of 500 by calling an azure function to do the processing.
500 (10 sec) x 500 (10 sec) = 250,000 (20 sec)
Since an Azure Function is just another HTTP request, you can assume triggering 500 of them will take ~10 seconds. Each function is triggered with a payload of 500 postbacks to execute. This gives us the ability to process 250k callbacks in ~20 seconds. Not too shabby!
So how do I use it?
Let’s take the example of sending an email. You have a function that is responsible for sending an email. You simply need two things: the function to be executed at the right time, and which email to send. Postbacks does this by allowing you to specify the request body you’d like included in your callback. Let’s take a look at how you’d trigger a callback using the Postbacks API.
POST https://api.postbacks.io/v1/requestPostback
As you can see, Postbacks makes scheduled tasks a breeze.
We love feedback, whether it’s your thoughts, concerns, feature requests, or use-cases, leave us a note below!
