saksh-queue-management

A queue management package using Bull and MongoDB with additional features like retry mechanism, priority jobs, delayed jobs, job progress, job events, job cancellation, and monitoring.

Installation

Install the package using npm:

npm install saksh-queue-management

Usage

Setting Up MongoDB Connection

First, set up the MongoDB connection string:

const QueueManager = require('saksh-queue-management');
require('dotenv').config();

(async () => {
    await QueueManager.sakshSetMongoDBConnection(process.env.MONGODB_URL);
})();

Adding a Job to the Queue

Add a job to the queue with optional settings:

const myQueue = new QueueManager('userQueue');

myQueue.sakshAddJob({ task: 'Task 1' }, { priority: 1, delay: 5000 });

Processing Jobs in the Queue

Define a callback function to process each job and handle retries:

const processTask = async (task, updateProgress) => {
    console.log(`Processing task with data:`, task.data);
    for (let i = 0; i <= 100; i += 20) {
        updateProgress(i);
        await new Promise(resolve => setTimeout(resolve, 200)); // Simulate async work
    }
    console.log(`Task with data ${task.data} completed`);
};

myQueue.sakshProcessJobs(processTask);

Handling Job Completion and Errors

Register callbacks for job completion and error events:

myQueue.sakshOnComplete((job) => {
    console.log(`Job ${job.id} completed successfully`);
});

myQueue.sakshOnError((job, err) => {
    console.error(`Job ${job.id} failed with error:`, err);
});

Retrieving Pending and Completed Tasks

Retrieve all pending and completed tasks from MongoDB:

(async () => {
    const pendingTasks = await myQueue.sakshGetPendingTasks();
    console.log('Pending Tasks:', pendingTasks);

    const completedTasks = await myQueue.sakshGetCompletedTasks();
    console.log('Completed Tasks:', completedTasks);
})();

Cancelling a Job

Cancel a job in the queue:

(async () => {
    await myQueue.sakshCancelJob('jobId');
})();

Running the Worker Script

Create a worker script (worker.js) to continuously process jobs:

const QueueManager = require('saksh-queue-management');
require('dotenv').config();

(async () => {
    await QueueManager.sakshSetMongoDBConnection(process.env.MONGODB_URL);

    const myQueue = new QueueManager('userQueue');

    const processTask = async (task, updateProgress) => {
        console.log(`Processing task with data:`, task.data);
        for (let i = 0; i <= 100; i += 20) {
            updateProgress(i);
            await new Promise(resolve => setTimeout(resolve, 200)); // Simulate async work
        }
        console.log(`Task with data ${task.data} completed`);
    };

    myQueue.sakshProcessJobs(processTask);

    myQueue.sakshOnComplete((job) => {
        console.log(`Job ${job.id} completed successfully`);
    });

    myQueue.sakshOnError((job, err) => {
        console.error(`Job ${job.id} failed with error:`, err);
    });
})();

Run the worker script using Node.js:

node worker.js

Or use a process manager like PM2:

npm install pm2 -g
pm2 start worker.js
pm2 logs worker

Monitoring and Managing Queues with Bull Board

The saksh-queue-management package integrates with bull-board to provide a web-based dashboard for monitoring and managing your queues.

Setting Up Bull Board

1. [Create a Server to Serve the Dashboard]

Create a new file named server.js to set up an Express server that serves the bull-board dashboard:

const express = require('express');
const { createBullBoard } = require('bull-board');
const { BullAdapter } = require('bull-board/bullAdapter');
const QueueManager = require('saksh-queue-management');
require('dotenv').config();

(async () => {
    // Set MongoDB connection string
    await QueueManager.sakshSetMongoDBConnection(process.env.MONGODB_URL);

    const myQueue = new QueueManager('userQueue');

    const app = express();
    const { router } = createBullBoard([
        new BullAdapter(myQueue.queue),
    ]);

    app.use('/admin/queues', router);

    app.listen(3000, () => {
        console.log('Server is running on port 3000');
        console.log('Bull Board is available at http://localhost:3000/admin/queues');
    });
})();

2. Run the Server

Start the server to access the bull-board dashboard:

node server.js

Open your browser and navigate to http://localhost:3000/admin/queues to view the dashboard.

Features of Bull Board

• Queue Overview: View all your queues and their statuses.
• Job Details: Inspect individual jobs, including data, logs, and errors.
• Job Actions: Retry, remove, or promote jobs directly from the interface.
• Real-Time Updates: Get real-time updates on job progress and status changes.

The saksh-queue-management package integrates with bull-board to provide a web-based dashboard for monitoring and managing your queues.

Features

• Retry Mechanism: Retries failed jobs a specified number of times.
• Priority Jobs: Allows adding jobs with different priorities.
• Delayed Jobs: Supports delayed jobs that start processing after a certain delay.
• Job Progress: Tracks and updates the progress of a job.
• Job Events: Emits custom events for job lifecycle events (e.g., job added, job started, job completed).
• Job Cancellation: Adds the ability to cancel jobs that are in the queue but not yet processed.
• Dashboard for Monitoring: Integrates with bull-board to monitor and manage your queues through a web interface.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Author

susheelhbti susheel2339 at gmail.com

Summary

This README.md file provides an overview of the package, installation instructions, usage examples, and details about the features. You can customize the author information and repository links as needed.