Procrastination like a boss with queues
Because I like to build proof of concepts, working on several projects a week, without having to care about the server, the architecture, and all those cool devops stuff, I usually use Heroku. Ok, it’s also because it’s free.
And because I’m a developer, so I like to try new stuff. In my last article, I talked about how to implement the magic links authentication system. To do so, we had to send emails after the creation of the user account. If we use an SMTP endpoint instead of using an external SaaS (why?), sending an email could be an operation that can take up to 1 second. And sometimes we need to send our user information to some third parties, or even notify our team on slack that we have a new user. All those stuff take time and in regards to user experience, it’s not something nice. But do we really need to do all this just right now? Could you wait a bit? Delegate this operation to something else?
The answer is clearly no. You can delegate this work and use queues to do it in a nice way. But which one should I take? A RabbitMQ or a tricky and free Redis?
It’s time to test!
To answer this question, let’s make a quick benchmark with a simple use case. We’ll make a node server that receives a request and puts an objet in a queue. Then, we will make a node worker that will just take the object from the queue, and write the id in a file.
And with the previous process, we will use Apache Benchmark and the following command to stress our queues
ab -n 10000 -c 10 http://localhost:3000/test
By doing it this way, we will compare RabbitMQ with its own node library and Redis with two implementation of queues that are Bull and Kue.
Bull implementation
We will start with an implementation using Redis because, for me, it’s the simplest to understand.
The following file is the implementation of the server that will receive the request and put a job in the queue. Quite simple.
And the following script just runs a simple worker that takes a job and then writes a line in a file.
When we run the benchmark, the server is really performant and 50% of our requests are served in 3ms.
Kue implementation
The implementation with Kue is quite similar to the one with Bull. But the result is a bit different.
And here is the result that we have.
We can already see that the response time has doubled.
Amqplib implementation
Here the implementation is really different. I took the samples given by the official RabbitMQ website and adapted them to fit the needs of this example.
First drawback of this implementation is that the official library doesn’t handle promises. Quite annoying to do everything with callbacks.
In this case, the message that we send in the queue has to be a Buffer, quite frustrating to convert everything every time.
Time to compare the results
With those three example, I’ve made a comparison between the moment when the job is stored in our queue and the moment when the job is done during the stress test. The following chart is a representation of the treatment time of a job while the number of calls increases. The response time of the solutions using Redis evolve exponentially while the solution using RabbitMQ stays stable in time.
What to choose ?
Based on these results, and considering the difference of price between RabbitMQ and Redis on Heroku, we can reflect around what we’ll use this queue for. If your queue is used to notify your services about all the events on your server and that your server handles thousands of requests by second, RabbitMQ might be the solution. But if you only use your queue to notify your service that send emails with a magic link, you probably just need a free Redis. If your website is a real success, you’ll still be able to increase the number of workers in order handle all the processes.
