Job Queue

A Node.js Job Queue library using Redis.

Features

• Create job queues
• Create workers to process jobs on those queues
• Store the queues and jobs in Redis for data persistence
• Use hooks to trigger actions during the job lifecycle

Dependencies

• Node.js
• Redis

Install

npm i @anephenix/job-queue

Usage

You will need a create a Redis client that is promisified with bluebird.

// Dependencies
const bluebird = require('bluebird');
const redisLib = require('redis');
bluebird.promisifyAll(redisLib.RedisClient.prototype);
bluebird.promisifyAll(redisLib.Multi.prototype);
const redisConfig = {};
const redis = redisLib.createClient(redisConfig);

Once you have that, you can create a queue like this:

const { Queue } = require('@anephenix/job-queue');

const emailQueue = new Queue({ queueKey: 'email', redis });

Adding jobs

Once you have the queue ready, you can add jobs like this:

const job = {
	name: 'job-001',
	data: {
		from: '[email protected]',
		to: '[email protected]',
		subject: 'Have you got the document for ML results?',
		body: 'I want to check what the loss rate was. Thanks.',
	},
};

emailQueue.add(job);

Setting up workers to process those jobs

Workers can be setup like this:

const { Worker } = require('@anephenix/job-queue');
const sendEmail = require('./sendEmail');

class EmailWorker extends Worker {
	async processJob(job) {
		this.status = 'processing';
		try {
			await sendEmail(job);
			await this.completeJob(job);
		} catch (err) {
			await this.failJob(job);
		}
		return;
	}
}

const emailWorker = new EmailWorker(emailQueue);

Workers are the base class on which to create Workers tailored to processing the job. In the example above, we have an EmailWorker whose processJob function is customised to send an email via the 'sendEmail' function. The worker is now setup to start processing jobs.

Starting the worker

await emailWorker.start();

The worker will now poll the queue for available jobs. Once it has one, it will take the job and process it.

Stopping the worker

await emailWorker.stop();

Advanced Features

Using hooks in the Queue

Hooks are a way to trigger functions before and after these actions are called on the queue:

• add
• take
• complete
• fail

This gives you the ability to do things like collect data on how many jobs are being added to a queue, how quickly they are being processed, and so on.

There are 2 types of hook, pre and post. A pre hook is called before the action is triggered, and a post hook is called after.

The way to setup hooks to call can be demonstrated in the example below:

const queueKey = 'email';
const queue = new Queue({
	queueKey,
	redis,
	hooks: {
		add: {
			pre: async job => {
				// Do something with the job before it is added
				return job;
			},
			post: async job => {
				// Do something with the job after it is added
				return job;
			},
		},
		take: {},
		complete: {},
		fail: {},
	},
});

License and Credits

©2021 Anephenix OÜ. Job Queue is licensed under the MIT license.