Queue

This node allows to queue messages and releases them time or event triggered.

During data processing it might be necessary to process a huge set of data, e.g. from a large csv input file, with many thousands of records. Further processing might have issues with such a big amount of data, so that it is useful to process this data sequentially in smaller blocks instead all at once. E.g. due to limitations in bulk operations on databases, or due to memory restrictions.

The standard split node might be no help, because it creates many parallel flows at once, which could overload the system, or takes up other ressources.

This queue node collects the input, but then drains smaller amounts as needed. Draining can occur automatically (time triggered), or manually (event triggered). So instead of processing big amount of data at once in parallel, smaller blocks will be released and the big volume of data serialized.

Nodes

• queue
  

Issues

All input is stored in memory. The node allowes to restrict the use of memory to some percentage (default is 90 %), so that a system crash due to lack of memory can be avoided.

However, if input of messages occurs faster than its draining, the queue will grow and finally use all available memory. If that happens an error will be triggered containing the overflow input. That input will not be queued. Use a catch node to fetch errors.

The node uses the node.js heap memory, which is used by all flows and thus all queue nodes.

Problems

To report issues or bugs, please send a note to [email protected].

License

Apache-2.0 License.