How we run background jobs in Node.js at scale
Setting up the Node.js Bull, Sidekiq Style
This article will help you set up Sidekiq style background job & cron processing capability in Node.js using Bull while keeping simplicity & scale at the centre.
Background: About Sidekiq
Anyone who has used Sidekiq in Ruby alongside Rails applications knows that it’s one of the greatest async, background processing & cron tools out there. It’s a fast, lightweight, and easy to set up background processing gem.
Below are the common use cases and Sidekiq provides out-of-the-box solutions for them
- Easy to interface cron jobs: Using Sidekiq + Sidekiq Scheduler, you can set up a bunch of cron jobs by describing cron schedule and executor workers.
- Triggering async workers: For triggering less critical comms/jobs from primary APIs in order to process them asynchronously (to reduce latency + compute). For triggering delayed jobs.
- Easy to set up side process: Run job execution as a separate process/service so that it can be managed & scaled independently. Job triggering services & execution services communicate over Redis.
- Web Console: To view, trigger, monitor jobs
Sidekiq takes care of all the queue setup, operations & wiring (You only have to specify queue names). It also gives you interfaces in your Rails code to trigger async jobs. That’s what makes it a great library.
Node.js Bull
If you search for similar packages in Node.js, you’ll find bull, node-cron, node-schedule amongst the top few. While node-schedule is more popular, it only provides cron solution. Bull is the most commonly used background queue processing tool. But it does not provide as easy and clean interfacing as Sidekiq and you have to set up a lot of things manually. You have to set up each queue independently and wire it in the processor service. So I’ll provide a step by step guide for each of the above use cases below.
The Setup
Setting up queues:
Set up all the queues, both for crons & async jobs in a single config file. In order to share queues between primary APIs/service (to trigger jobs) and processor service (to execute jobs), you have to initialise the queue in both services. Duplicating this is prone to error.
So a practice we’re following
- Initialise & export all queues (crons & async jobs alike) in a config file
- Define their processing worker functions mapped with queues
- Import the queues in primary service (APIs) to trigger jobs (“the service”)
- Import and “process” all the queues in the processing service (background processor, “the processor”)
The Queues Config
All queues are defined in a config file. This config will be shared between the service & the processor.
The Usage
The exported queues will be used by the service to trigger jobs:
The Processing
The queues map will be for setting up the processor:
Now we have seen the basic job processing and cron set up. Adding new crons and queues is fairly easy. You have to define new queues in the list & set their worker functions.
Running as a standalone process:
node bull_processor.js Running this will start a new process that listens to and processes all queues.
Scaling
These processes run independently. They can be run in a docker container as well for easy & fast scaling. WE can spin up 10s, 100s of processes/containers pretty quickly as per the needs. You can also publish the queue lengths & use metrics/alarms to auto-scale your machines/containers easily with ECS, EKS, Kubernetes, etc based on the queue lag.
The basic setup is done here. The below code is for a special use case.
Async jobs and workers: (the uncommon ones)
We’ve seen how to set up queues and trigger jobs. These are useful when you have common use cases, like sending notifications/comms, run cleanup job, etc. But what if you want to just run any function asynchronously? Setting up queues, workers every time is a pain!
There’s a convoluted way to set this up. You can have a dedicated queue for all such jobs & choose functions dynamically.
Other useful options
That’s all! Hope this helps you set up and get your application up and running quickly.
