Efficient design for daemons tasks
The majority of backend applications run continuously in background some asynchronous tasks like sending mail, synching with external resources, importing some data from external api… Over time it become a real deal with your infra resources, for example, if you have 10 background tasks running permanently, each of them have to load an entire framework plus some linked libraries on memory, it’s depends on framework certainly but your are about ~80Mo — ~100Mo+ for each task, that’s mean for 10 tasks your consuming 1Go of memory reservation just for daemons!
Ajouter une description
In cloud domain when every small thing has a price you have to calculate carefully and efficiently your costs, otherwise you will be faced with an expensive invoice or even the worst, a service interruption and I would not wish it on anyone.
I think that the simplest solution is to not run permanently your daemons, instead run them on demand. The classic way is to keep a script running permanently (while true) checking DB each x seconds to give a real time feeling to the end user. I don’t think that is the right way, why not to run our tasks only when we need them to run.
Modern architecture — From Daemon to task
To do that let’s place a “Message Broker” like RabbitMQ and a service Consumer implemented in your preferred programming language. The Consumer turns without stop to check permanently RabbitMQ queue, this is the unique daemons that we will have in this approach. Secondly we will have to modify some parts of our backend to send “create message” to RabbitMQ each time we want to launch an asynchronous task.
Real world example
Using the classic way to handle send mail tasks, we need a background script that will checks DB every second for a ready to send mails, some times it gets 0 ready mail, some times it gets several.
NB: heavy job like send mail must be asynchronous because of connexion to SMTP, sending mail content (perhaps attachment), and waiting for SMTP response.
The modern way as like as the classic way, we must have a background script with a single role (send mail) except that it must be sleeping (stoped) by default. When a user create a new account, our system must send him a confirmation mail. To accomplish that we will create a message in our Message Broker so our Consumer can find the message in the queue and starts “send mail script”. No matter if we receive 1000 account creations at the same time, once “send mail script” has started it will continue until it handles all of the 1000 operations and it will go to sleep (stoped) agin.
Ajouter une description
Benefits
- Memory optimization: We consume memory/resources just when we had to do. All the other tasks are stoped, we will launch them when the associated events are happened. No need to keep all daemons running.
- No watch dog: No worries about restarting the daemon after a OOM or an other reason where the kernel kill the process to survive.
- DB connections optimization: No reason to check DB every second to ensure a real time execution, if an event is happened the task will be started as fast as the classic method do.
