Background workers in Blitz

Everyone wants a fast and performant web application. As the functionality of your application grows you will likely integrate with many services for things like sending email, billing, analytics, etc. This functionality usually comes with network IO which can be both painfully slow and error prone.

When a user registers for your application and you want to send them a confirmation email. Do you really need to wait for the email to send to show them the registration success page? What happens if the email fails to send?

Background workers are a great solution to these types of problems. Typically a background worker is a long running service that is waiting for work to appear in some queue. When your service adds a new task to the queue the worker is able to complete it independently of the service that enqueued it. This means instead of waiting for an HTTP request to a third party to complete all you will need to do is wait for the job to be enqueued in Redis. If the third party service is down and an exception is thrown then the job will automatically be retried instead of being lost forever.

Alternatively, you might just have long running processes because there is simply a lot of work to do. Maybe you are transforming images, processing video files, or training a machine learning model.

Regardless of what you are doing it is often nice to be able to push work to a background worker and leave your web server for handling what it was meant to do (serving responses to web requests!).

Want to read this story later? Save it in Journal.

With Blitz utilizing Javascript for both the client and server it only makes sense to also use Javascript for your background workers. Think about all that sweet sweet code re-use.

One of the leading frameworks for background workers in the Javascript ecosystem is BullMQ and it’s what we’ll be using to get background workers running with Blitz.

In this article we will walk through setting up a BullMQ queue and worker to your Blitz application in a way where you can re-use all of your existing code in your background workers.

To get started we will need to add BullMQ to the project, which we can do using yarn like so:

yarn add bullmq

Because BullMQ requires long running worker processes we need a way to bundle all of our source code into a script ready to be run by Node. We will use Webpack to get this done but your favorite bundler will work just as well.

We will now add Webpack and a couple of other packages we will need to our project:

yarn add -D webpack webpack-cli webpack-node-externals ts-loader typescript dotenv

webpack and webpack-cli will let us actually run the Webpack binary to build our BullMQ worker script.

webpack-node-externals allows us to easily tell Webpack to ignore our node_modules directory. When running node we do not need to bundle our node_modules directory as Node is able to resolve modules there itself.

typescript and ts-loader allow Webpack to compile and bundle the typescript we have in our Blitz project. If you aren’t using Typescript in your project then you won’t need these packages.

dotenv lets us inject any environment variables the BullMQ workers will need via a .env file.

How exactly you structure everything inside of your Blitz repo is up to you but we will be putting everything inside of the jobs directory that blitz creates for us by default.

Inside this directory we create three subdirectories:

workers — organizes all of our workers
queues — organizes all of our queues and queue names
src — entry point for our worker process

Let’s start by creating our first queue for sending asynchronous emails. This is an example of what ourjobs/queues/index.js file might look like:

import { Queue } from "bullmq"
export const EMAIL_QUEUE_NAME = "sendemail"
export const queueConfig = {
  connection: { 
    port: parseInt(process.env.REDIS_PORT, 10), 
    host: process.env.REDIS_HOST 
  },
}
export const emailQueue = new Queue(EMAIL_QUEUE_NAME, queueConfig)

We import the Queue class from BullMQ library. We use a constant to define the name of the queue to make it easier to update in the future if we need to. We then define the Redis connection options (host and port) derived from env variables. Lastly, we instantiate a new queue given the name and the connection options defined above.

We export all of these items because we will need them later in both our Blitz app and our worker process.

The email worker in jobs/workers/emailWorker.tsx is similarly straight forward:

import { Worker } from "bullmq"
import { EMAIL_QUEUE_NAME, queueConfig } from "jobs/queues"
import { sendEmail } from "app/mailer"
const emailWorker = new Worker(
  EMAIL_QUEUE_NAME,
  async (job) => {
    return sendEmail(job.data)
  },
  queueConfig
)
export default emailWorker

We import the Worker class from BullMQ library, our email queue name and config from the queues file we just created. Lastly, we import a wrapper function for actually sending an email via Sendgrid API.

We then create a worker instance by passing in the name of the email queue, the function to run, and the Redis configuration we defined in our queues file.

In the entry point file jobs/src/index.tsx all we need to do is import our worker:

import { emailWorker } from "jobs/workers"

In the Sendgrid mailer wrapper we can push a job onto the queue like this:

import { emailQueue } from "jobs/queues"
emailQueue.add("sendgrid", options)

We import the queue we created earlier and then add the job along with the email options the worker will need to send the actual email. The first parameter to add is the name of the job. We can set this to be anything we like. This gives us the ability to split work based on queue or job name.

Now we need to configure Webpack to bundle this source and run our workers in a Node process. We can create a config file in our Blitz root directory named worker.webpack.config.js :

const path = require("path")
const nodeExternals = require("webpack-node-externals")
const projectRoot = path.resolve(__dirname)
module.exports = {
  target: "node",
  mode: process.env.NODE_ENV,
  context: projectRoot,
  entry: {
    worker: "./jobs/src/index.tsx",
  },
  resolve: {
    modules: ["node_modules", projectRoot],
    extensions: [".ts", ".tsx", ".js", ".json"],
  },
  externals: [nodeExternals()],
  module: {
    rules: [
      {
        test: /\.tsx?$/,
        loader: "ts-loader",
        options: {
          configFile: "tsconfig.worker.json",
        },
      },
    ],
  },
}

A couple things to note with the configuration:

entry make sure this points to our worker entry script. Eventually we will want to separate workers into separate processes to be able to scale them based on their individual needs. We can easily add multiple worker scripts here to be bundled by Webpack when we need this functionality.

externals is an array to exclude from bundling directly into your output file. This is where we can utilize the webpack-node-externals package to easily exclude the node_modules directory from being bundled.

module this is where we can define loaders and plugins for processing specific file types. We use ts-loaderto enable support for Typescript. The config file tsconfig.worker.json is shown here:

{
  "extends": "./tsconfig.json",
  "compilerOptions": {
    "noEmit": false
  }
}

We extend the default Blitz/Next tsconfig.json but need to change the noEmit option from true to false because Next.js doesn’t need to actually emit the Javascript file.

Lastly, we need to setup some new npm scripts in our package.json to build and start our new worker.

"build:worker": "yarn run webpack --config ./worker.webpack.config.js",
"start:worker": "node ./dist/worker.js",
"start:worker:dev": "node -r dotenv/config ./dist/worker.js"

To build the worker we define thebuild:worker script where we run Webpack and pass in our worker specific configuration file.

To start the worker we just need to run the built file with Node. We can add a separate script for starting it in development that uses dotenv package to load our environment variables from a .env file.

At this point we should be good to go!

Try building the worker script using yarn build:worker and then running it using yarn start:worker:dev. Once the script is running try enqueuing a job by triggering whatever flow you setup to add a job to your queue. The job should get picked up and executed by our worker.

I hope you found this article helpful for getting background workers running alongside your Blitz application.

📝 Save this story in Journal.

👩‍💻 Wake up every Sunday morning to the week’s most noteworthy stories in Tech waiting in your inbox. Read the Noteworthy in Tech newsletter.