Integrating BullMQ with NextJs

With the latest versions, Next.js has become a viable full-stack framework that can be used to build complex and scalable web applications.

When building fullstack applications and working with API routes, Route handlers (in Next.js 13+) or Server actions (in Next 13.4+), we may need to perform asynchronous tasks. For that we can use a queue system like BullMQ with Redis.

BullMQ is a Node.js library that provides a simple and reliable way to implement Redis-based queues in Node.js applications.

It also provides a number of advanced features such as job retries, job progress tracking, and concurrency control.

Installing bullmq within a Next.js application

First, install BullMQ and Redis within the Next.js project.

npm install bullmq ioredis --save
# or yarn add bullmq ioredis

Next, we have to create a worker script which will pull the tasks from a queue and process them.

Create workers/sample.worker.ts file within the Next.js project. The workers folder can be inside the src directory if you’re using the Next.js src project structure; otherwise, you can put it in the project root folder.

// src/workers/sample.worker.ts

import { Worker, Queue } from 'bullmq';
import Redis from 'ioredis';
const connection = new Redis(process.env.REDIS_URL!);

export const sampleQueue = new Queue('sampleQueue', {
    connection,
    defaultJobOptions: {
      attempts: 2,
      backoff: {
        type: 'exponential',
        delay: 5000,
      },
    },
  });

const worker = new Worker(
  'sampleQueue', // this is the queue name, the first string parameter we provided for Queue()
   async (job) => {
    const data = job?.data;
    console.log(data);
    console.log('Task executed successfully');
  },
  {
    connection,
    concurrency: 5,
    removeOnComplete: { count: 1000 },
    removeOnFail: { count: 5000 },
  }
);

export default worker;

Next, we can modify the package.json file to add a new script to run this worker. We also need to install a few additional packages if they are not already installed.

npm install dotenv tsx --save
# or yarn add dotenv tsx

// package.json
{
  ...
  "scripts": {
    ...
    "worker:sample": "dotenv -e .env.local -- npx tsx --watch src/workers/sample.worker.ts"
  },
  ...
}

Make sure to provide the correct path for the worker script. In our case, it was created as src/workers/sample.worker.ts.

Running the worker

npm run worker:sample
# or yarn worker:sample

Keep the terminal open and let’s continue to add some jobs to the queue.

Adding jobs to the queue

Now we can add new jobs to the queue within our Next.js server-side code (e.g., API routes, route handlers, and server actions).

If you’re using Next.js 13+:

// src/api/add-job.ts
import { NextResponse } from 'next/server';
import {sampleQueue} from '@/workers/sample.worker';

export async function GET(){
  const data = {
     // any serializable data you want to provide for the job
     // for this example, we'll provide a message
     message: 'This is a sample job'
  }
  await queue.add('someJob', data);
  return NextResponse.json({'status': 'Message added to the queue'});
}

If you’re using previous Next.js versions:

// /pages/api/add-job.ts
import { NextApiRequest, NextApiResponse } from 'next';
import {sampleQueue} from '@/workers/sample.worker';

export default async (req: NextApiRequest, res: NextApiResponse) => {
  const data = {
    // any serializable data you want to provide for the job
    // for this example, we'll provide a message
    message: 'This is a sample job'
  }

  await queue.add('someJob', data);
  res.status(200).json({'status': 'Message added to the queue'});
};

You can see a new job being created and executed by the worker when you visit http://localhost:3000/api/add-job URL.