Queueing with BullJS

Introduction

A queue is a data structure that places elements in a sequence, similar to a stack. The difference is the methodology of accessing the data. It operates using the LIFO concept, meaning the last item to be added to the stack will be the first one to get accessed whereas the queue operates using FIFO meaning the first item inserted will be the first one out of the queue.

Please find the illustrations below:

Queue (Photo by wikipedia)
Stack

There is a multitude of ways Queues are used in developing the most common being Sequential execution of tasks/jobs. It allows for developers to add jobs on one end and after processing leaving on the queue on another end. It usually operates on FIFO (First In First Out) methodology, which means the first job to arrive in the queue will get executed early before moving on to the next job arriving in the queue. After execution, the job is removed from the queue on the other end.

What queues are good for:

  1. It allows for breaking down large amounts of jobs into smaller more manageable jobs.
  2. It gives you the ability to add multiple processors for a particular job. Meaning we can attach multiple consumers that can pick up jobs from the queue and execute them
  3. More complex workflows can easily be visualized and implemented with multiple queues ensuring consistency and atomicity.

BullJS

BullJS is one of the many node packages that can be used to implement queues in your application. Like any other Queueing library, it utilises the FIFO methodology.

npm install bull

Bull is extremely fast in execution and much of it owes to the fact that it uses Redis as the underlying technology that backs it up. This means data generated or inserted into the queue is stored in memory for faster reads and writes. Redis also supports a pub/sub mechanism that aids in receiving callbacks for failed, completed or failed jobs.

Example implementation for Queues using BullJS

There are two parts to this implementation. A Producer, one that will produce the jobs to be added to the queue and a Consumer whose task is to execute the tasks added by the producer.

A Consumer should inform the Producer once the job has completed execution.

First, define the Redis configuration

const defaultRedisConfig = {
redis: {
host: redisHost,
port: redisPort,
db: redisDatabaseNumber,
},
};

Consumer logic

// Redis server Configuration 
const <queue-name> = new Bull("<queue-name>", defaultRedisConfig);
setQueues([
new BullAdapter(<queue-name>),
...
]);
// Consumer processing logic
<queue-name>.process(async (job) => {
// Processing logic
});

Producer logic

const Bull = require('bull', { redis: defaultRedisConfig });
const <queue-name> = new Bull('tigo-queue');
<queue-name>.add(<Job Data>);

For similar blogs to this please checkout the ClickPesa Engineering Blog on:

Medium

Dev.to

Hashnode

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store