How to Set Up Queue Jobs in NextJS Using BullMQ

How to Set Up Queue Jobs in NextJS Using BullMQ

Posted By

kamlesh paul

on

Dec 10, 2024
Copy PasteQueue

In this guide, we will walk through how to set up queue jobs in NextJS using BullMQ, a powerful and scalable Redis-based queue library. This approach is useful for handling background jobs like sending notifications or processing orders without blocking the main thread.

We previously covered how to set up Next.js cron jobs without Vercel using BullMQ, so this article builds upon that by showing how to leverage the same library for queue jobs. By using BullMQ, you can efficiently handle background processes without the need for additional packages or dependencies.

Table of contents

Prerequisites

Make sure you have the same prerequisites as mentioned in our previous article on cron jobs without Vercel:

  • Redis installed and running locally or on a remote server.
  • Next.js set up in your project.
  • The BullMQ package installed.

Folder Structure

To keep things organized, we’ll structure our files as follows:

/server
  ├── redis.ts
  ├── /queues
  │     ├── index.ts
  │     ├── config.ts
  ├── /jobs
        ├── processOrder.job.ts
        ├── processUserNotification.job.ts

This structure helps in separating the Redis connection, job configurations, and the actual job processing logic.

Step 1: Redis Configuration

First, we’ll set up the Redis connection using the ioredis package.

server/redis.ts
import Redis from 'ioredis';
 
export const redisConnection = new Redis('redis://127.0.0.1:6379', {
  maxRetriesPerRequest: null,
  enableReadyCheck: false,
});

This connects your Next.js app to the Redis instance running locally at 127.0.0.1:6379. You can update the Redis URL based on your environment.

Step 2: Setting Up the Queue

We will dynamically import all job files from the /jobs folder:

server/queues/index.ts
import { readdirSync } from 'fs';
import { join } from 'path';
 
const jobsPath = join(__dirname, 'jobs');
const jobFiles = readdirSync(jobsPath);
 
jobFiles.forEach(file => {
  import(join(jobsPath, file));
});

This ensures that all job processors are automatically loaded when the app starts.

Step 3: Queue Configurations

Here, we define the default job options, such as retry attempts and backoff strategies:

server/queues/config.ts
import { DefaultJobOptions } from "bullmq";
 
export const defaultQueueConfig: DefaultJobOptions = {
  attempts: 3,
  backoff: {
    type: 'exponential',
    delay: 1000,
  },
};

This setup retries failed jobs up to 3 times with an exponential backoff.

Step 4: Defining Jobs

Now, let’s define two job processors: one for processing orders and one for sending user notifications.

server/queues/jobs/processOrder.job.ts
import { redisConnection } from "@/server/redis";
import { Queue, Worker } from "bullmq";
import { defaultQueueConfig } from "../config";
 
const queueName = 'processOrder';
 
interface QueueInterface {
  orderId: number;
  title: string;
  amount: number;
}
 
const processOrderQueue = new Queue(queueName, {
  connection: redisConnection,
  defaultJobOptions: {
    ...defaultQueueConfig,
    delay: 500,
  }
});
 
new Worker(queueName, async (job) => {
  const data: QueueInterface = job.data;
  console.log('Processed order:', data);
}, {
  connection: redisConnection
});
 
export const addToProcessOrderQueue = (data: QueueInterface) => {
  return processOrderQueue.add(queueName, data);
};

This worker processes order-related jobs. It listens for incoming jobs, processes them, and logs the result.

server/queues/jobs/processUserNotification.job.ts
import { redisConnection } from "@/server/redis";
import { Queue, Worker } from "bullmq";
import { defaultQueueConfig } from "../config";
 
const queueName = 'processUserNotification';
 
type QueueInterface = {
  userId: number;
  message: string;
};
 
const processUserNotificationQueue = new Queue<QueueInterface>(queueName, {
  connection: redisConnection,
  defaultJobOptions: {
    ...defaultQueueConfig,
    delay: 500,
  }
});
 
new Worker<QueueInterface>(queueName, async (job) => {
  const data: QueueInterface = job.data;
  console.log('Processed user notification:', data);
}, {
  connection: redisConnection
});
 
export const addToProcessUserNotificationQueue = (data: QueueInterface) => {
  return processUserNotificationQueue.add(queueName, data);
};

This worker handles sending notifications to users.

Step 5: Triggering Jobs from the Frontend

  • We can create a simple UI to trigger these jobs. Here’s how you can do it:

Trigger Job UI

app/(dashboard)/TriggerJob.tsx
'use client';
 
import { Button } from '@/components/ui/button';
import { processOrder, processUserNotification } from '@/server/actions/triggerJobs.action';
import React from 'react';
 
export default function TriggerJob() {
  return (
    <div className='flex flex-col gap-5 border-2 border-dashed p-3 border-black'>
      <Button variant={'destructive'} onClick={() => processOrder().then(res => alert(res.message))}>Process Order</Button>
      <Button variant={'destructive'} onClick={() => processUserNotification().then(res => alert(res.message))}>Notify User</Button>
    </div>
  );
}

Backend Logic to Trigger Jobs

server/actions/triggerJobs.action.ts
'use server';
 
import { addToProcessOrderQueue } from "../queues/jobs/processOrder.job";
import { addToProcessUserNotificationQueue } from "../queues/jobs/processUserNotification.job";
 
export const processOrder = async () => {
  await addToProcessOrderQueue({
    orderId: 1,
    title: 'Iphone 16',
    amount: 10000
  });
 
  return {
    success: true,
    message: 'Order processed'
  };
};
 
export const processUserNotification = async () => {
  await addToProcessUserNotificationQueue({
    message: 'Test notification',
    userId: 1
  });
 
  return {
    success: true,
    message: 'User notified'
  };
};

Using the Trigger in the Main page

  • To test the functionality, simply include the TriggerJob component in your main layout or any page:
MainPage.tsx
import TriggerJob from '@/server/actions/TriggerJob';
 
function MainPage() {
  return (
    <div>
      <TriggerJob />
    </div>
  );
}

TriggerJob-console.webp

Conclusion

By following this guide, you can set up queue jobs in Next.js using BullMQ. This approach is flexible and can be expanded to handle various background tasks, from processing orders to sending notifications. Feel free to adjust the job configurations and queue logic to suit your specific needs.

For the full source code of this implementation, you can check out the following repository: Next.js Role and Permission with Queue Jobs.

For more advanced BullMQ features like job prioritization, rate limiting, or delayed jobs, check out the BullMQ documentation.

Share this article

122 views