Workers and Node: kue it up.

This is going to be a quick write up on how to use kue to facilitate workers in node. I won’t claim to be an expert, but if you are here reading this, neither are you. 😜 Hence, if you have something that can help us both improve, please fire away in the comments.

Workers of the wrong kind

Motivation

Firstly, let’s understand the motivation behind using workers in any application (especially, node). Sometimes, we find ourselves with tasks that can be taken out of the current request-response cycle. A typical example is generally that of asynchronous tasks such as sending emails. “Bah! I’m a node developer. We breathe async.”, said the ignorant node developer. Here it becomes important to understand the difference between web workers and deferred results or promises.

JavaScript is single threaded, so, you cannot use promises to run code asynchronously — once the code runs that fulfills the promise, no other code will run. This really proves tricky when you want that code to do some heavy lifting. Plus, you can’t just fire and forget a promise in the name of asynch.

Is this right to you?

You need to handle failure cases and retries. Imagine that email was for a delivery confirmation. You will have to resolve the promise successfully and accordingly retry sufficient times to ensure delivery. That implies resolution logic to include this. AND kaboom, your single threaded application is now handling a lot of heavy logic that doesn’t necessarily need to sit inside it.

Kue

Kue is a priority job queue backed by redis, built for node.js. (Copied for the benefit those too lazy to open its github). Include it in your project by doing

npm install kue --save

Incase you don’t have redis set up, follow this link to get you up and running.

But you gotta put in work, work, work, work, work, work, work

So, I’ll throw in a worker here that leverages kue to get its jobs. Let’s call it emailWorker.js. Kindly go through it once. I’ve picked up possible points that might need explanation afterwards.

emailWorker.js

It may be a little difficult to bring yourself to read the comments but I’ve tried to keep them as verbose as possible. (Yeah I know, we don’t write comments, then how can we ever expect someone to read them. Stupid mortals.)

Something I deliberately skipped out from explaining in detail was the job.attempts(0 , ()=>{}) bit. The attempts syntax is to offer some control over the job execution. This will become clearer once we write the logic for pushing a job in the controller.

A job is considered completed (success or failure both) once the done() method is invoked. Kue maintains jobs in 4 states: active, inactive, failed or complete. Active is the state of jobs currently being processed. Inactive state is when a job has stopped execution and might need intervention to complete. Handling of this is pretty important because this blocks one of the concurrent processing threads. You can imagine the world of pain it will cause once all concurrent jobs are blocked. Completed and failed states are exactly what they sound like.

Next we change our controller logic to assign the asynch task to the worker.

goodAsynch.js

So let’s look at what happened here. We create a job and push it onto the queue with the tag ‘email’ and pass the data as an object. We have a worker that is listening to all jobs of type ‘email’ and hence, this will find itself getting executed by that worker.

.removeOnComplete(true) is responsible of removing the job from the queue altogether once it has been executed. We can always choose not to do this but this is just idle memory being used up in redis. If you want to keep this data, you can always mould your worker logic to persist this data in the DB.

.attempts(5) defines the maximum number of retries the job will go through before it decides to give up on it. Remember when we changed this parameter inside the worker? It just prevented these 5 tries from happening as soon as the request was a fishy one. Because if a request returns say a 404 not found, it means it probably will never get through. Same as someones affection for me. #FML #selftroll

.backoff({delay: 60*1000, type:’exponential’}) sets the gap between retries. By default the next retry happens immediately after a failure. Normally exponential is a good choice since it provides sufficient time between attempts for common issues like ratelimiting etc to settle down. You can read more about it here.

Finally, save() persists the job in the queue.

The jobs listen to events using redis’ pubsub model. Hence various event triggers can be handled inside the code itself. This isn’t blocking and gives good control incase the queue starts to fail. emailJob.on(‘failed’, ()=>{}) does just that.

Deploying these

A more robust convention is to use supervisor to deploy. But a simpler way is to just add the workers to your pm2 file as another app. I’m using the first app from the previous article I wrote.

boot_workers.js is just a small file that will transpile the code from the ES6 syntax.

require('babel-register');
require('emailWorker.js');

All good if you ask me. Kindly post comments if you feel something needs more explanation or any other inputs that can help me improve.

Cheers 🍻!

--

--

Tryharding to keep tech quirky

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store