node-etoile

node-etoile is a workflow engine for nodejs. The design principle was quite thin: create an easy-to-learn distributed system-ready agnostic solution.

Yes, it must be as agnostic as possible allowing to extend basic types, replace subsystems with ease not put a big integration-burden to your shoulders. It is designed to be enterprise-ready, to be convenient for the enormous and highly complex distributed workflows of T-systems working with huge time-fragmentation.

To be true to this principle, the system is introduced by short notes. ;)

Features

• simple as 1: only the business logic is needed to be written. Nothing else, no hard logic, structure, interaction, storage, transaction-management...
• runtime deployment: entities are read and deployed from the 'deploy' folder at runtime.
• agnostic design: all conceptual types are defined via prototypes. Étoile can be extended and alter as your imagination drives your. Respects embedder systems.
• versioning: an entity might appear in multiple instances with different versions to serve all steps of evolution of your solution as a representation of backward compatibility.
• fully async communications trustworthily to the nodejs /js world
• built-in transaction management: all workflows partly or entirely are traceable/re-executable.
• built-in storage management via MongoDB
• built-in messaging bus integration via AMQP
• built-in Websockets-based services for publishing communication between entities
• built-in rest-based services for gathering information about the states of the running engine

Index of the page

• Installation
• Set up
• HelloWorld.js
• Concept
• Configuration
• First tutorial

Installation

npm install node-etoile 

Set-up

Simplest case.

1. Define and place your entities to the 'deploy' folder

2. In your system the etoile object needs to be loaded.

   var etoile = require('./etoile');

3. Initial configuration needs to be passed.

   etoile.load( [ 'config/etoile.json' ] );

4. Service instances needs to be passed.

   etoile.instigate( );

5. Let the system work. If shutdown is required, call this:

   etoile.shutdown();

That's it!

HelloWorld.js

function Bob() {
	Bob.super_.call( this, 'Bob' );
}
global.util.inherits(Bob, global.etoile.prototypes.Provider);
var bPrototype = Bob.prototype;
bPrototype.greetings = function(id, params) {
	this.respond( id, null, 'Hello World!' );
};
var bob = new Bob( ).activate( );

A simple workflow entity providing a simple service: greetings. It is automatically published and available to everybody. Notwithstanding that all communication is async, the transaction is automatically closed and everything is stored and logged in the background.

Concept

Communication

Most basic term defining the structure sent between the points of a workflow. This is the internal communication data format of Étoile. Defines basic attributes about the caller, the recipient, parameters, answer and circumstantial properties like timestamp.

All communication is async in Étoile and built upon the event system of nodejs.

Note: every communication needs to be answered or it will considered as a lost message harming the execution of your workflow!

Entity

It is the most basic term. Supertype of everything defining basic functionality like:

init, respond, send

Every entity is activated by the calling of the function init() by the system's internal services. During this service, all methods defined by the programmer will be found and published as business logic services. All incoming communication is passed through these methods.

If a service function needs to call in another entity or respond an ongoing communication, the respond and send functions are used.

Bus

Messaging bus of Étoile. The common communication interface for every entities, all comm go through this point. Just a dispatcher dealing with routing, version handling, etc..

Provider

This is the spot for you. The representation of the workflow points. All business logic will be implemented here.

The function activate is needed to be called - like the "hello world" example shows above - by you with optional passing the necessary configurations as well.

By calling this function, the publishing process will be started and all necessary internal management will be accomplished.

Firestarter

Just a wrapper around the provider to make it lighter and free all management functionality. Fully transparent to the programmer, no attention is needed to deal with it.

Chronicler

This is the storage entity of the system. The bus is using this object to store every dispatched communication and manage the transactions. Étoile provides an in-memory solution as default, but a built-in mongoDB-based is also available or feel free to extend with your own solution.

Interface

Interfaces aimed to define the entry-points of this system. If a workflow can be initiated by call from another system via HTTP Post or AMQP message, this type needs to be specialized.

Étoile provides an AMQP-based and a REST-based built-in interface if you need it, but feel free to extend by any protocol/standard you need to use.

Packager

You might need to deal quite often with inherited data formats and protocols. A legacy system might binds your work painfully sometimes. Despite of the interface, the format of incoming messages might vary in a wide spectrum beyond your jurisdiction to consider it.

This type is a data format converter, converting the incoming packages - dispatched from the interfaces - to the internal Communication data format and vice versa.

Index

Configuration

The configuration of Étoile is very straightforward: a simple json file given to the load function of the étoile instance. See chapter Set up.

(Note: The filepath does not need to be given if the default values fits your needs.)

Let's look over the content of this file.

Server settings

"server": {
	"port": 8080,
	"apiKeys": [ "849b7648-14b8-4154-9ef2-8d1dc4c2b7e9" ],
	"discoverPath": "discover",
	"protoPath": "proto",
	"context": "/api",
	"socketPath": "/bus"
}

This optional element enables the internal server for restful-, and websockets-based services including required API keys if needed as for security reasons. For details please check the connect-rest's documentation.

Deploy settings

"deploy":{
	"folder": "./deploy/",
	"requirePrefix": "../../deploy/"
}

This optional element enables the auto runtime deployment service of Étoile. The folder where the javascript files will be read out and a computed parameter for the code loader to identify the context.

Log settings

"log":{
	"consoleMode": true,
	"level": "debug",
	"path": "logs/etoile.log",
	"error": "logs/error.log"
}

If consoleMode is true, every logs will be directed into the console, otherwise the give filepaths will be used as output. The level sets what minimum level you want to see in the output.

Let me emphasize, that you can pass a bunyan logger instance to the load function of etoile object if you want to use your own already existing logger object.

Additional settings

It might happen to need to configure customized entities like specialized interfaces or chronicler instance. In this case, you might want to separate these settings for better manageability and readability as follow:

etoile.load( ['config/etoile.json', 'config/mongodb.json', 'config/amqp.json'] );

This will config the Étoile itself and the mongoDB-based built-in chonicler and the built-in amqp-based interface as well in separate config files.

Index

First tutorial

I guess you know more then enough to code finally. So after having checked out the git and - now for simpler case - presuming that default configuration is used, you are ready to code. :)

To implement a workflow all you need to do is to create specialized Providers.

Every Provider should be a separated file and put to the deploy folder.

So at the end, you will have a bunch of JS code in that folder deployed automatically by Étoile, and ready-to-serve. So get ready! :)

Let's take a really simple workflow - basically the code can be found in the test folder.

Scenario:

• Alice wants to go out for a dinner. Therefore asks Bob to reserve a table for today
• Bob - by receiving the allocation request, let Alice know about the ongoing work on this topic. Alice is getting bored. ;)
• Bob asks the restaurant "Sailor's Palace" (SP) if there is a table free at that time
• SP might responses back: 'Free'
• Bob reserve a table at SP if there is any free there
• When SP acknowledges the reservation tells Alice about the successful reservation
• Alice then happily goes there with BOB

Not an oversophisticated scenario but enough for introduction. We have 3 entities, so we have 3 files

bob.js, alice.js and sailorsPalace.js

Alice

function Alice() {
	Alice.super_.call( this, 'Alice' );
}
global.util.inherits(Alice, global.etoile.prototypes.Provider);
var aPrototype = Alice.prototype;
aPrototype.feelHungry = function(id, params) {
	this.send(id, 'Bob', null, 'reserveTable', {date: Date.now()}, function( id, report, err, response ){
		if(report)
			this.logger.info('Im getting unpatient now...');
		if(err)
			this.send(id, 'Bob', null, 'figureOutSomething' );
		if(response)
			this.logger.info('All right, I go with You.');
	});
};
var streamA = new Alice( ).activate( );

Simple enough. Alice asks Bob to reserve a place. if a report comes from Bob - saying he is working on - alice is getting unpatient, in case of error Alice forces Bob to figure out something. In case of a successful response, the dinner is up.

About reporting: report is something which is highly recommended to use in case of time-consumming interactions. Sometimes a business flow might have communication with duration of days.

So in an abstract way, a report must be sent from the recipient back to the sender telling that the request is being processed.

Do not forget, that every communication needs to be answered with error or by providing some normal output!

"What are those ids over there?" If you consider a rather complex workflow, where an entity is involved into multiple parallel transactions and in a given transaction is involved into multiple interactions, it is good to know a reference point where the current flow is standing. That id is a reference about the ongoing interaction sent or received by that given entity.

If in your workflows, it has no meaning, just ignore it...

Sailors Palace

function Sailor() {
	Sailor.super_.call( this, 'Sailor' );
}
global.util.inherits(Sailor, global.etoile.prototypes.Provider);
var sPrototype = Sailor.prototype;
sPrototype.checkDate = function(id, params) {
	this.respond( id, null, 'Free.' );
};
sPrototype.reserveTable = function(id, params) {
	var self = this;
	setTimeout( function (){
		self.respond( id, null, 'Table reserved.' );
	}, 500 );
};
var streamA = new Sailor( ).activate( );

This entity provides 2 services: checkDate and reserveTable invoked by Bob as described above. I took the liberty to put a delay in the reserveTable to add some time-distance in the service. Bob has to wait for the result of the reservation request.

Bob

function Bob() {
	Bob.super_.call( this, 'Bob' );
}
global.util.inherits(Bob, global.etoile.prototypes.Provider);
var bPrototype = Bob.prototype;
bPrototype.figureOutSomething = function(id, params) {
	this.logger.debug('Ouch!');
};

bPrototype.reserveTable = function(id, params) {
	this.report( id, 'I\'m on it!' );
	this.setState( {date: params.date, alice: id} );

	this.send( id, 'Sailor', null, 'checkDate', params, function( id, report, err, response ){
		if( response )
			this.send( id, 'Sailor', null, 'reserveTable', this.getState('date'), function( id, report, err, response ){
				if( response )
					this.respond( this.getState('alice'), null, 'Accomplished.' );
			});
	});
};
var streamA = new Bob( ).activate( );

For JS-addicted eyes, it is very easy to read when is happening what. But there is more behind the scenes.

Why not this?

bPrototype.reserveTable = function(id, params) {
	this.report( id, 'I\'m on it!' );
	this.send(id, 'Sailor', null, 'checkDate', params, function( id, report, err, response ){
		if( response )
			this.send(id, 'Sailor', null, 'reserveTable', params, function( id, report, err, response ){
				if( response )
					this.respond( id, null, 'Accomplished.' );
			});
	});
};

Let's think about it. The engine must provide state-safety during the whole workflow. It means, that states need to be persisted and read back preparing ourself to a unwanted case when system crashes while an entity is sending message to another one. The time when the closure will be executed might be months later including several server restart, so this very adorable nature of closure - meaning to reach the variables of the embedder function - cannot be used.

Unfortunately there is not tool in JS to retrieve the available environmental variables for a closure. You can bind a function to an object as a "this" reference but cannot reach the param variable from it in this example. For that very reason, one has to use the state-related structure of the Providers.

Every entity has an associated state object for every transaction in progress. This is persisted and managed by the chronicler object in the background. Why bother then? Reason is simple. The content of this state is depending on your logic.

From abstract design aspect, you must consider that this workflow must be rock-solid even if the server suffers from operational issues. This means, that the callback functions might not be executed by the same uptime period for example.

Not mentioning the distributed design goal. It might happen, that the response processing in a given interaction is made by an entity on a different node in a cluster.

This also leads you here, that you cannot rely on the parameters which are available for that closure provided by the embedder function(s).

This is where you have to design the state of entity 'Bob' following the most precious rule: every interaction must be executable independently from each other in time and space.

This is why Bob storing the date and the identifier of the comm with alice as a state and retrieves when needed. Everything else is managed by the engine behind the scenes.

Note: You might observed the this in the closure. This is working in a response processing function because those functions will be bound to the given entity before it is called.

"IS THAT IT???". Yep. It is that simple. You can build-up any workflow with these tools. Believe me, workflows at T-systems with really high complexity and time-distance need to be managed, and this tool is the right thing we have to use.

How to use it:

save those files in the deploy folder and execute the followings if the etoile engine is running already:

global.etoile.firestarters['Alice'].providers[global.etoile.AllVersion].feelHungry( global.puid.generate() );

If you do not have any interface deployed. Or any specific analogue communication if you have.

Index

Custom bunyan logger

Custom storage level

Custom interfaces

Connect server

Architecture

Deeper concept