sqs-priority-consumer

Build SQS-based applications without the boilerplate. Just define a function that receives an SQS message and call a callback when the message has been processed.

This is branched from sqs-consumer, but has a number of fixes to keep long-processing messages alive and to support switching between multiple queues in order to support a "priority" queue.

Installation

npm install sqs-priority-consumer --save

Usage

const Consumer = require('sqs-consumer');

const app = Consumer.create({
  queueUrl: ['https://sqs.eu-west-1.amazonaws.com/account-id/priority-queue-name', 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name'],
  waitTimeSeconds: [20, 5],
  sticky: [1000000000, 20000],
  
  handleMessage: (message, done) => {
    // do some work with `message`
    done();
  }
});

app.on('error', (err) => {
  console.log(err.message);
});

app.start();

• The queues are polled continuously for messages using long polling.
• Messages are deleted from the queue once done() is called.
• Calling done(err) with an error object will cause the message to be left on the queue. An SQS redrive policy can be used to move messages that cannot be processed to a dead letter queue.
• By default messages are processed one at a time – a new message won't be received until the first one has been processed. To process messages in parallel, use the batchSize option detailed below.

Credentials

By default the consumer will look for AWS credentials in the places specified by the AWS SDK. The simplest option is to export your credentials as environment variables:

export AWS_SECRET_ACCESS_KEY=...
export AWS_ACCESS_KEY_ID=...

If you need to specify your credentials manually, you can use a pre-configured instance of the AWS SQS client:

const Consumer = require('sqs-consumer');
const AWS = require('aws-sdk');

AWS.config.update({
  region: 'eu-west-1',
  accessKeyId: '...',
  secretAccessKey: '...'
});

const app = Consumer.create({
  queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
  handleMessage: (message, done) => {
    // ...
    done();
  },
  sqs: new AWS.SQS()
});

app.on('error', (err) => {
  console.log(err.message);
});

app.start();

API

Consumer.create(options)

Creates a new SQS consumer.

Options

• queueUrl - Array - The SQS queue URL. If an array is provided, the URLs will be cycled
• waitTimeSeconds - Array - The time to wait for each queue. If just a number is provided, the same time will be used for all queues.
• sticky - Array - When a message is pulled from a queue, keep repolling this queue for sticky milliseconds. This is good if queue #1 has 5000 messages in it, but queue #2 has 0 messages. If you don't specify sticky, it will process a message from queue #1, then poll queue #2 with a waitTime of 1 second. It will then move back to queue #1. If you want a queue to be processed repeatedly without checking other queues, you can set this number to be big.
• region - String - The AWS region (default eu-west-1)
• handleMessage - Function - A function to be called whenever a message is received. Receives an SQS message object as its first argument and a function to call when the message has been handled as its second argument (i.e. handleMessage(message, done)).
• attributeNames - Array - List of queue attributes to retrieve (i.e. ['All', 'ApproximateFirstReceiveTimestamp', 'ApproximateReceiveCount']).
• messageAttributeNames - Array - List of message attributes to retrieve (i.e. ['name', 'address']).
• batchSize - Number - The number of messages to request from SQS when polling (default 1). This cannot be higher than the AWS limit of 10.
• visibilityTimeout - Number - The duration (in seconds) that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request.
• terminateVisibilityTimeout - Boolean - If true, sets the message visibility timeout to 0 after a processing_error (defaults to false).
• waitTimeSeconds - Number - The duration (in seconds) for which the call will wait for a message to arrive in the queue before returning. If more than one queueUrl is provided, you can provide an array here. The duration will be applied to the matching queueUrl
• sticky - Array - If more than one queueUrl is provided, you can choose to have some queues "sticky". This means that if it finds a message, it won't move on to the next queue for the next poll. This is useful if you want to have a priority queue.
• authenticationErrorTimeout - Number - The duration (in milliseconds) to wait before retrying after an authentication error (defaults to 10000).
• sqs - Object - An optional AWS SQS object to use if you need to configure the client manually

consumer.start()

Start polling the queue for messages.

consumer.stop()

Stop polling the queue for messages.

Events

Each consumer is an EventEmitter and emits the following events:

|Event|Params|Description| |-----|------|-----------| |error|err, [message]|Fired when an error occurs interacting with the queue. If the error correlates to a message, that error is included in Params| |processing_error|err, message|Fired when an error occurs processing the message.| |message_received|message|Fired when a message is received.| |message_processed|message|Fired when a message is successfully processed and removed from the queue.| |stopped|None|Fired when the consumer finally stops its work.| |empty|None|Fired when the queue is empty (All messages have been consumed).|

AWS IAM Permissions

Consumer will receive and delete messages from the SQS queue. Ensure sqs:ReceiveMessage and sqs:DeleteMessage access is granted on the queue being consumed.