Table of contents

• Description
• Dependencies
• Application Configuration
• Application Bootstrapping
• Module Configuration
• Module Bootstrapping
• Application Flow Architecture
• Pre-requisites
• Setup
  • Install
    • Install Application
    • Install Module
• License

Description

This project provides a set of components to help build applications using TotalJS Flow. The two main components are the Application and the Module. The Application is responsible for processing requests and managing the modules. The Module is responsible for handling all the operational logic. The TotalJS Flow project is a great tool for developing applications using flow based programming. When building larger applications where inter-module communication, multiphase processing, and scaling need to be taken into account it can be difficult to manage. This project provides a set of components to facilitate the building of larger applications using TotalJS Flow. Both the Application and the Module need to communicate with a redis server. Redis is used as the backend for how Application and Module instances communicate with each other. The Application component needs to communicate with a MongoDB server. The MongoDB server is primarily used to store details about the modules and their status. This is used so that the Application instances can be scaled out horizontally and keep in sync when it comes to processing.

Dependencies

• TotalJS Flow
• Redis Server
• RethinkDB Server

Application Configuration

• Application Name (Required: String)
• Processing Mode (Required: String)
  • "Everything" (all) = Both input and API requests are processed
  • "Input" (input) = Only input requests are processed
  • "API" (api) = Only API requests are processed
• Redis Connection (Required: String)
• RethinkDB Connection (Required: String)

Application Bootstrapping

• Application does sanity checks on the configuration
  • Check connection to MongoDB and Redis
• Then application executes the "init" phase.
  • Initialize the Application Worker
  • Initialize the Module Manager
  • Create a new FlowMessage (with config as data) then send it to the "init" output and store message data on 'end' as Application Data (AppData)
• Then application executes the "boot" phase.
  • Boot the Application Worker
  • Boot the Module Manager
  • Create a new FlowMessage then send it to the "boot" output with AppData and update message data on 'end' as AppData
• Then application executes the "start" phase.
  • Start the Application Worker
  • Start the Module Manager
  • Create a new FlowMessage then send it to the "start" output with AppData and update message data on 'end' as AppData
• Then application executes the "ready" phase.
  • Ready the Application Worker
  • Ready the Module Manager
  • Create a new FlowMessage then send it to the "ready" output with AppData and update message data on 'end' as AppData
  • Start the status monitor which sends status to UI at interval
• The application is now ready to process requests.

Module Configuration

• Label (Required: String)
• Name (Required: String)
• Application (Required: String)
• Mode (Required: String)
  • "Everything" (all) = Both input and API requests are processed
  • "Input" (input) = Only input requests are processed
  • "API" (api) = Only API requests are processed
• Redis Connection (Required: String)

Module Bootstrapping

• Module does sanity checks on the configuration
  • Check connection to Redis
• Then module connects to the application and registers itself.
  • Module sends a AppRegisterMessage to the application informing it of its existence.
  • Application responds with a AppRegisterResponse.
• If AppRegisterResponse.success is true, then the module is registered.
  • Module executes the "init" phase.
    • Initialize the Module Worker
    • Create a new FlowMessage (with config as data) then send it to the "init" output and store message data on 'end' as Module Data (ModData)
  • Module executes the "boot" phase.
    • Boot the Module Worker
    • Create a new FlowMessage then send it to the "boot" output with ModData and update message data on 'end' as ModData
  • Module executes the "start" phase.
    • Start the Module Worker
    • Create a new FlowMessage then send it to the "start" output with ModData and update message data on 'end' as ModData
  • Module executes the "ready" phase.
    • Ready the Module Worker
    • Create a new FlowMessage then send it to the "ready" output with ModData and update message data on 'end' as ModData
    • Start the status monitor which sends status to UI at interval
    • Start heartbeat monitor which sends heartbeat to application at interval
• If AppRegisterResponse.success is false, then the module is not registered.
  • Module is disabled.

Application Flow Architecture

• Application Input (Input -> Phases -> Output)
  • Application Input is the entry point for the application which is the input labeled "input".
  • Application Input is then sent to the "phases" output with the result being an array of strings used for that inputs processing phases.
  • Application Input metadata is sent to each module requesting its involvement (phases and priority).
  • Application Input is processed by each module involved according to priority with the results becoming the updated Application Input.
  • Resulting Application Input is sent to "output" output for final processing.
• Application Message (Message -> Application -> Response)
  • Application Message is the message that is sent to the application from a module.
  • Application Messages are sent to the application via the BullMQ job queue system.
  • AppRegisterMessage is sent to the application to register a module.
  • AppUnregisterMessage is sent to the application to unregister a module.
  • AppHeartbeatMessage is sent to the application to inform it that a module is still alive.
  • AppAPIMessage is sent to the application to execute an API call (Handled by "api" output. When the message ends the data is sent back as the result).
  • AppModuleMessage is sent to the application to execute a module API call.
  • Application Messages are sent to the "api" output with the result being sent back to the module as an AppMessageResponse.
• Module Message (Module <-> Application <-> Module)
  • Module Message is the message that is sent to a module from the another module through the Application.
  • Module Messages are sent to the application via the BullMQ job queue system.
  • The application sends the message to the appropriate module for processing and the result is sent back to the requesting module.

Pre-requisites

• Node.js >= 8.16.0
• NPM >= 6.4.1 (NPM comes with Node.js so there is no need to install separately.)

Setup

Install

Install Application

• Create a new TotalJS Flow.
• Create a new component and containing the contents of Application Component.
• Add the Application component under the "Application" group to the Flow.
• Attach desired input and outputs to the Application component.
• Configure Application component.

Install Module

• Create a new TotalJS Flow.
• Create a new component and containing the contents of Module Component.
• Add the Module component under the "Module" group to the Flow.
• Attach desired input and outputs to the Module component.
• Configure Module component.

Messages

Application Messages

Theses are messages sent TO the application.

License

MIT License © Colton McInroy