Setting Up Bull to Create Queues with Heroku Redis: Easy Solution

Bull for Node.js with Heroku Redis: How To

Hi, my name is Evan and I am the lead developer at Array Assistant. A few months ago, we put out our very first version of our product. It was shaky but it worked. However, as we moved more towards a real launch, we have been aiming at creating new tools with even better functionality.

For a brief intro to what we do, we create plugins that allow you to use AI in Microsoft Office, specifically Microsoft Excel. To do this, I use a Heroku server running Node.js web dynos to get the information from the AI to the client on Microsoft Excel.

While upgrading, one of the main issues we were running into was that the infamous Heroku H12 timeout error. This error means that any HTTP request taking over 30 seconds will automatically timeout. There is a simple fix though, switching the longer tasks over to workers. To do this, you need to implement a ‘job queue’, often done with Redis.

Heroku has a Redis implementation that works well within the program, but the documentation is a little shaky, especially when using Bull. Bull is a job queue manager that allows you to easily add jobs to a queue, track their status, retrieve their final value, and delete them from the queue when you’re done. It makes life a lot easier but it can be difficult to setup. Below I will show you how I set it up.

Getting Redis in Heroku

Before we start I should mention that almost everything I do in this article will be either code or done through the CLI, as it makes things a little more universal.

First, start by opening a terminal or CLI into your Heroku app. Then, you can add the Redis addon like so:

heroku addons:create heroku-redis:mini -a your-app-name

The above code will provision a mini redis database for your app. From here, you will also need to install bull in your Node server project. Do this with the following command:

npm install bull

Heroku will automatically provide you will all the necessary environment variables to setup your database in your server environment. Therefore, there is no need to even leave the project. However, if you are curious, you can find more information about your Redis database by going to the web dashboard for you Heroku app, selecting the ‘Resources’ tab, and finding the Redis database below ‘Add-ons’.

Setting Up a Bull Queue (No TLS)

For testing and running small servers, the mini tier of Redis on Heroku should be enough. In this case, creating a queue is simple as no TLS is needed.

First, simply import Bull into your server where you are planning to send jobs to the worker, this may be your index or your router, and also add it to your new worker file.

To require is in Node, simply import Bull like so:

const Queue = require('bull');

Now, you are going to want to get the Redis url for your database. However, this should already be in your Heroku environment variables. To get this value, I pulled it like this:

const REDIS_URL = process.env.REDIS_URL;

Now that we have both of the ingredients, we can create our work queue. The name you choose for your queue does not matter, but it should stay the same between your worker and index files. Create a queue like this:

const workQueue = new Queue('jobList', REDIS_URL);

Now, you can follow the documentation to add, process, retrieve, and delete jobs from your queue using Bull. If you decide to upgrade your Redis database to ‘premium-0’ or higher, you will need to authenticate with TLS. To do this, follow the steps below:

Setting Up a Bull Queue (with TLS)

When you upgrade, another level of security is required. However, since this instance of Redis is run through Heroku, the two servers are essentially running together. Therefore, instead of authenticating with TLS like usual, we instead just create a work around.

First, we will get our Bull instance imported into the index and worker files and then grab our Redis url from our environment variable. The code will look similar to before:

const Queue = require('bull');
const REDIS_URL = process.env.REDIS_URL;

Next, we need to create our workaround for the new security measure. If you are using Redis 6.0.0 or higher, then we use the following. We create a new variable to hold our queue options:

const queueOptions = {
  redis: {
    socket: {
      tls: true,
      rejectUnauthorized: false,
    },
  },
};

If you are using Redis 6.0.0 or lower, we use the following.

const queueOptions = {
  redis: {
    tls: {
      rejectUnauthorized: false,
    },
  },
};

Finally, we set up the queue similar to before, except with add in our options at the end:

const workQueue = new Queue('workList', REDIS_URL, queueOptions);

This will tell the Redis instance that it doesn’t need to worry about the extra security layer, therefore allowing us to bypass it.

Conclusion

I hope this helps out. I know when I was starting with this that the Bull, Heroku, and Redis documentation was confusing me. I tried a number of different solutions but none of them worked in the correct way. This will work for any version of Redis that is connected through Heroku.

If people want, I can also create an article explaining how to setup Bull queues and use them a little more in depth. However, as the documentation was so poor for this, I decided that there need to be an article out there talking about it.

If you like this article, consider following me or, more importantly, checking out my new business. To do so, you can simple click here: Array Assistant. Once again thanks for reading. If you see any issues with this article, leave you thoughts in the comments below!