node-red-contrib-persistent-queue

A contributed Node-RED node, that queues undeliverable messages to file for later delivery.

This node is most valuable for Node-RED installations that have unreliable network connectivity. Messages that can't be delivered, are instead queued on disk until communications are again restored.

This is a fork from node-red-contrib-msg-queue. It fixes some issues and implement file size limit for the DB file. When this limit is reached, the new messages will be forwarded to the next node even if the Internet connectivity is lost, preventing the disk from being filled up.

Overview

If an outbound Node-Red node, such as an MQTT node goes offline, subsequent messages are lost until the node reconnects. By wiring a Queue node ahead of the outbound node, and wiring status messages into Queue, undeliverable messages will instead be queued to a local file.

When the outbound node reconnects, Queue will forward on messages from the queue in the same order they were received, until the queue becomes empty or the outbound node again goes offline. While Queue drains the queue of messages, any new messages that arrive are appended to the queue to ensure ordered delivery is maintained.

Once the queue becomes empty, and the outbound node is still connected, Queue will again forward on messages and bypass the queue storage entirely.

Installation

Install node-red-contrib-msg-queue by following the adding nodes instructions from the Node-RED Getting Started Documentation.

The following instructions use npm at the command line.

cd $HOME/.node-red
npm install node-red-contrib-msg-queue

Usage

To use the node, launch or re-launch Node-RED (see running Node-RED for help getting started).

An example-flow.json is available that matches the illustration in the overview. You can copy and paste this flow into Node-RED and tinker to get a feel for how it works.

Or if you prefer, read the following explanation and view the screen shot.

Queue Configuration

The Connected Status Matches section is a regular expression ^connected which matches all status text messages that start with connected. In addition to regular expressions, you can also specify a simple substring match. Because there is no value specified to match Disconnected Status Matches, any message that doesn't match ^connected will be deemed a disconnected state.

Conversely, if a Disconnected Status Matches value is provided, but not for Connected Status Matches, then any status text that does not match will be deemed a connected state.

Finally, if both a Connected Status Matches and Disconnected Status Matches value is given, then any status text doesn't match either, will be ignored, and the current state will remain.

You must also specify a filename for the sqlite database that stores queued messages during a disconnected state. This database file must not be shared between multiple queue nodes.

TODO

A TODO List of possible future features is included. Contributions welcome.

Licence

Copyright (c) 2017 Damien Clark, Damo's World Licenced under the terms of the GPLv3

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DAMIEN CLARK BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Acknowledgements

Like others who stand on the shoulders of giants, I'd like to acknowledge the contributions of the following people/groups without which, more directly, this modest Node-RED node would not be possible.

• Creators of the queue icon
• Nick O'Leary and Dave Conway-Jones for creating Node-Red