MsgFlo - Flow-Based Programming with Message Queues

Implementation of the Flow-Based Programming using message queues as the communications layer between different processes. Initial message queue transports targeted are AMQP and MQTT.

MsgFlo lets you build robust polyglot FBP systems spanning multiple computers/devices. A node can be implemented in any language, to reuse existing code, libraries and developer know-how.

In FBP each component is a black-box that processes and produces data, without knowledge about where the input data comes from, or where the output data goes. This ensures that a service is easy to change, and facilitates automated testing.

MsgFlo is designed to enable partial and gradual integration into existing systems; by using standard broker/transports, not placing restrictions on message payloads, allowing to use existing queue names, and integrating non-MsgFlo nodes seamlessly.

Status

In Production

• Used in production at TheGrid website builder, with AMQP/RabbitMQ. 20 roles, 1'000'000 jobs/weekly+
• Used in production in imgflo image processing server. 4 roles, 200'000 jobs/weekly+
• Used for IoT networks at hackerspaces c-base and Bitraf, using MQTT/Mosquitto.

Client support

• msgflo-nodejs makes it easy to make Node.js participants in JavaScript/CoffeeScript
• noflo-runtime-msgflo makes it super easy to use NoFlo in the participants
• Basic support for C++ participants with msgflo-cpp and MicroFlo
• Basic support for Python participants with msgflo-python
• Basic support for browser participants with msgflo-browser
• Basic support for Arduino participants with msgflo-arduino
• Experimental support for Rust participants with msgflo-rust

Tooling

• msgflo executable implements the FBP runtime protocol.
• Initial support for automated testing using fbp-spec
• Experimental support for visually building networks using Flowhub
• guv provides autoscaling of workers when using Heroku/AMQP.

Licence

MIT, see ./LICENSE

Documentation

Please refer to https://msgflo.org

Support

MsgFlo is a part of Flowhub, a platform for building robust IoT systems and web services. We offer an Integrated Development Environment and consulting services.

Debugging

The msgflo executable, as well as the transport/participant library uses the debug NPM module. You can enable (all) logging using:

export DEBUG=msgflo*