Knetik Node Engine (KNE)

This is documentation for the Knetik Node Engine, this document should cover everything you need to know for project setup and development with KNE.

1. Overview
2. Project Setup
3. Submodules
4. Modules
5. Configuration
6. Logging
7. Project Creation
8. Tests

Overview

The Knetik Node Engine is a modular server written in Javascript with NodeJS.

KNE was built with three concepts in mind:

1. Be modular with configurable and reusable components.
   • KNE is built to be divided into easily configured submodules and modules. These are similar but both serve a different purpose.
   • The submodules and modules use a simple dependency tree to specify what should be loaded and used at runtime.
2. Scale horizontally and vertically with minimal configuration.
   • For vertical scaling, KNE was built with a master/worker architecture using Node's cluster module.
   • For horizontal scaling, KNE was coded to work in conjunction with other instances that it is load balanced with.
3. Bridge the gap between JSAPI and client specific feature implementations.
   • We wanted to avoid cluttering JSAPI with client specific components, so KNE is used to fill in the blanks.
   • KNE is also used to fill in features that JSAPI does not currently support, such as websockets.

General notes to remember when developing with KNE:

• Modules and submodules should attempt to stick with the concepts listed above.
• When a worker encounters an error, it will shutdown and the master process will spawn a new worker.
• When the master process encounters an error, the entire process and its workers will shutdown. (This is why it is important to keep as much logic/code on the workers as possible.)

Project Setup

How to install an existing project that uses KNE:

1. Ensure you have read the project specific README.
2. Pull project repository branch.
3. Install NodeJS v4.2.1 and NPM v3.8.0.
4. Depending on what submodules the project uses you may also need to have other dependencies installed, see the project specific README for details.
5. Set the environment variable NODE_ENV to either "development" or "production". *See Note
6. CD into project directory and run "npm install".
7. Run "node app.js" to generate a configuration file, you may need to prefix this with your NODE_ENV setting.
8. Edit config.js in configs/ folder, most defaults can be left for development purposes but any configuration set as undefined must be set.
9. Run "node app.js" again.
10. KNE should now be running properly.

*Note: If you are running KNE in development mode then you must either install Bunyan npm install -g bunyan or set the environment variable KNE_DISABLE_BUNYAN=true.

Submodules

The core of KNE is divided into multiple submodules that can be reused between modules and other submodules.

Submodules differ from modules in that they are always included with KNE and implement a core feature that most modules will need to reuse, such as a connection to a database.

Using Submodules

Though submodules are always included with KNE, they will not be initialized if they are not marked as a dependency by other modules/submodules. You will not be able to use a submodule if it is not specified as a dependency and initialized.

You can depend on a submodule and make use of it like so:

//Constructor for your module.
function MyModule() {
    var self = this;
}

//Get the Mongo submodule from the engine instance.
MyModule.prototype.init = function() {
    var kneMongo = this.engine.getSubmodule('kne-mongo');
    //Make some database calls...
};

...

//Require the Mongo and the unreleated JSAPI submodule.
module.exports.submodules = [
    'kne-mongo',
    'kne-jsapi'
];

Submodules are only initialized once per worker, and have the option to be initialized on the master for special cases.

Creating Submodules

Submodules can be created and configured in virtually the same way that a module can with a few differences.

NOTE: If you are creating a project with KNE chances are that you will not need to create a submodule. You will most likely need to create a module instead. Submodules are reserved for core functionalities.

General notes to remember when creating a submodule:

• Your submodule should try to follow the naming convention already in place. Ex. kne-my-module
• Submodules can depend on other submodules, but they cannot depend on external modules that are not included in KNE.
• Submodules will not be initialized unless they are required by a module or submodule that is being initialized.
• Functions in submodules should be a prototype (MySubmodule.prototype.exampleFunction) so that they can be overridden if needed, unless they are specifically meant to be private.
• Submodules must have an init function to be initialized, more on this is in the example below.
• Submodules are loaded in the order specified in app.js with the array submodulesAvailable. You must require your submodule in this array.
• Upon initialization, modules are given an reference to the engine this.engine and an instance of a Bunyan logger this.log.

An example of a simple submodule that will be initialized per worker:

//Submodule constructor.
function MySubmodule() {
    var self = this;
}

//The init function that is called to once the submodule has been created. It takes in a promise that must be resolved to continue starting the engine.
MySubmodule.prototype.init = function(defer) {
    try {
        //Attempt to initialize here.
        //...
        
        defer.resolve(); //If everything goes well, resolve the promise.
    } catch(e) {
        defer.reject(e); //Oh no an error, reject the promise so that KNE can handle the worker issue.
    }
};

//This function is called after all submodules have been initialized.
MySubmodule.prototype.postInit = function() {
    //Do some post initialization action.
};

//Export the submodule.
module.exports = MySubmodule;

//Define configurations for this submodule, more on this is available in the Configuration section. This can be omitted if not needed.
module.exports.configs = {
    test: 'testconfig'
};

//Require other submodules. This can be omitted if not needed.
module.exports.submodules = [
    'kne-mongo',
    'kne-jsapi'
];

Submodules can also be initialized on the master process or have a function that is ran on the master.

To initialize a submodule on the master process include this line:

module.exports.initMaster = true;

To run a function on the master process include something like this:

module.exports.initMaster = function(defer, engine, log) {
    try {
        //Perform master logic... 
        defer.resolve();
    } catch(e) {
        defer.reject(e); //Oh no an error! This will shutdown the engine.
    }
};

Submodule List

kne-express

A web server module built with Express.

This module will load any file in your module's routes/ folder that ends in .route.js.

Example route file:

//owner - The instance of the module this is contained in.
//engine - The instance of the engine.
//app - The Express application to use routing with.
module.exports = function(owner, engine, app) {
    app.get('/api/hello', function (req, res, next) { //Documention for using app can be found on Express's site.
        res.send('Hello there!');
    });
};

Example configuration in your projects config.js:

'kne-express': {
    'port': 8000
}

kne-jsapi

A module that initializes a JSAPI token for use between workers. This makes use of Restler.

How to use:

var jsapiClient = self.engine.getSubmodule('kne-jsapi').getClient(); //Get the JSAPI client from the module.
jsapiClient.get('users').then(function(res) { //Make a call to the users list endpoint that returns a promise.
    console.log('Users:', res);
}, function(err) {
    console.error('JSAPI error:', err);
});

Example configuration in your projects config.js:

'kne-jsapi': {
    'url': '[URL HERE]',
    'clientId': '[CLIENT ID HERE]',
    'clientSecret': '[SECRET KEY HERE]',
    'allowTokenRefreshOnAuthenticationMiddleware': true,
    'logLevel': 0 //The higher the level, the more verbose logging is.
}

kne-mongo

A module that initializes a Mongo connection, loads module models, and executes various other boilerplate code. Makes use of Mongoose for models.

This module will load any file in your module's models/ folder that ends in .model.js.

Example model file:

var mongoose = require('mongoose');

//owner - The instance of the module this is contained in.
//engine - The instance of the engine.
module.exports = function(owner, engine) {
    mongoose.model('Notification', mongoose.Schema({ //Documentation for creating a Mogoose model can be found on their site.
        identifier: String,
        userId: Number,
        tsCreated: Date,
        tsViewed: Date,
        viewed: {type: Boolean, default: false},
        data: mongoose.Schema.Types.Mixed,
        expires: {type: Date, expires: 1, default: null}
    }));
};

Example configuration in your projects config.js:

'kne-mongo': {
    'url': 'mongodb://localhost/kne-default'
}

kne-queue

A module that initializes a job queue in Redis. Implemented with Kue.

How to use:

//...
var queue = self.engine.getSubmodule('kne-queue').getQueue(); //This should be in your module's init function.
queue.process('my-module:myJob', self.myJob.bind(self)); //Documentation for this can be found on Kue's Github page.
//...

VideoService.prototype.myJob = function(job, done) {
    //Do some job things.
};

Example configuration in your projects config.js:

'kne-queue': {
    'redisUrl': 'redis://127.0.0.1:6379',
    'uiPort': '3000'
}

kne-redis

A module that initializes a connection with a Redis server. Uses NodeRedis for handling the connection.

Example configuration in your projects config.js:

'kne-redis': {
    'url': 'redis://127.0.0.1:6379'
}

kne-websocket

A module that initializes a websocket server built with Socket.IO.

How to use:

var kneWebsocket =  self.engine.getSubmodule('kne-websocket');
var socketClient = kneWebsocket.getClient();

socketClient.on('connection', function(socket) { //More documentation is available on Socket.IO's site.
    console.log('Client connection!');
});


//The JSAPI module has authentication middleware that can be used, so that connecting to the socket requires a JSAPI token.
var kneJsapi = engine.getSubmodule('kne-jsapi');
kneWebsocket.addMiddleware(kneJsapi.authenticationMiddleware.bind(kneJsapi));

Example configuration in your projects config.js:

'kne-websocket': {
    'port': 14000
}

kne-mysql

A module that initializes a connection pool to a MySQL server. Built with https://github.com/mysqljs/mysql.

NOTE: When you get a connection from the connection pool you must ensure you release it after it is used.

How to use:

var kneMysql = self.engine.getSubmodule('kne-mysql');

//Get a connection from the connection pool.
kneMysql.getConnection().then(function(connection) {
    connection.query('SELECT * from users', function(err, rows) {
        connection.release(); //ALWAYS release the connection, or subsequent queries will run out of connections.
        
        if (err) {
            defer.reject(err);
        } else {
            defer.resolve(rows);
        }
    });
}, function(err) {
    defer.reject(err);
});

Example configuration in your projects config.js:

'kne-mysql': {
    'host': '127.0.0.1',
    'port': 3306,
    'user': 'root',
    'password': 'mysql',
    'database': 'mysql',
    'connectionLimit': 2,
    'options': {}
}

kne-libraries

A module that initializes a associative array named lib on each app submodule holding any exported module in the submodule's lib directory

How to use:

Create a directory named lib inside your submodule directory. Place a module *.js file inside. Nested files are supported. The directory and file naming stucture inside lib is up to you.

Setup modules/app-service/lib/helpers/helper.js

module.exports = function(owner, engine) {
    class Helper {
        static titelize(str) {
            return str.replace(/\w\S*/g, function(txt) {
                return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();
            });
        }
    }

    return Helper;
}

Usage

const Helper = this.engine.getSubmodule('app-service').lib.get('helpers.helper');
const titleCasedString = Helper.titleize('this string');

Modules

KNE projects are divided into modules. Modules can be generic standalone repositories that KNE projects can require in their package.json or project specific located in their projects modules folder.

With project specific modules, the functionality is specific to your project and is not reusable. As for generic modules, the module's functionality should be reusable and would be put into a repository separate from your project's.

Using Modules

Modules will not be initialized if they are not added to your projects dependency list. Generic modules can be used by including their repository in your package.json and then adding their name to your dependency list. Project specific modules can just be put into the modules folder in the root of your project and then their folder name put into your dependency list.

You can depend on submodules and/or other modules and make use of them like so:

//Constructor for your module.
function MyModule() {
    var self = this;
}

//Get the Mongo submodule and a custom module from the engine instance.
MyModule.prototype.init = function() {
    var kneMongo = this.engine.getSubmodule('kne-mongo');
    //Make some database calls...
    
    var myModule = this.engine.getModule('prj-my-other-module');
    //Do some cool things with another module...
};

...

//Require the Mongo submodule and another module MyOtherModule
module.exports.submodules = [
    'kne-mongo',
    'prj-my-other-module'
];

Modules are only initialized once per worker, and have the option to be initialized on the master for special cases.

Information on submodules and their uses can be found in the Submodules list section.

Creating Modules

Each module in your project should strive to perform a single task. A good example would be the Notification Service, it only handles notifications and nothing more. (It is also a good example of a generic module.)

General notes to remember when creating a module:

• Modules should not be put into the KNE repository, if they are generic they should be placed in their own repository or if they are project specific they should be in your projects repository in the modules folder.
• Your module should try to follow the naming convention already in place. Generic Ex. kne-my-module Specific Ex. prj-my-module, where "prj" is shorthand for your project name.
• Modules will not be initialized unless they are in the dependency list created on initialization for your project.
• Functions in modules should be a prototype (MyModule.prototype.exampleFunction) so that they can be overridden if needed, unless they are specifically meant to be private.
• Modules must have an init function to be initialized, more on this is in the example below.
• Modules are loaded in the order specified in the dependency list created on initialization for your project.
• Modules should strive to not be dependent on other modules.
• Upon initialization, modules are given an reference to the engine this.engine and an instance of a Bunyan logger this.log.

Modules should follow this file structure:

prj-my-module/  - Project directory, the module will be specified/loaded with this name.
    src/    - Folder for module source code.
    models/ - Folder for Mongoose database models, only needed when using the kne-mongo submodule. More information under the Submodules List section.
    routes/ - Folder for Express routes, only needed when using the kne-express submodule. More information under the Submodules List section.
    app.js  - The file that is loaded by the engine for initiallization.

An example of a simple module's app.js that will be initialized per worker:

//Module constructor.
function MyModule() {
    var self = this;
}

//The init function that is called to once the module has been created. It takes in a promise that must be resolved to continue starting the engine.
MyModule.prototype.init = function(defer) {
    try {
        //Attempt to initialize here.
        //...
        
        defer.resolve(); //If everything goes well, resolve the promise.
    } catch(e) {
        defer.reject(e); //Oh no an error, reject the promise so that KNE can handle the worker issue.
    }
};

//This function is called after all modules and submodules have been initialized.
MyModule.prototype.postInit = function() {
    //Do some post initialization action.
};

//Export the submodule.
module.exports = MyModule;

//Define configurations for this module, more on this is available in the Configuration section. This can be omitted if not needed.
module.exports.configs = {
    test: 'testconfig'
};

//Require submodules. This can be omitted if not needed.
module.exports.submodules = [
    'kne-mongo',
    'kne-jsapi'
];

Modules can also be initialized on the master process or have a function that is ran on the master in the same way that submodules can.

To initialize a module on the master process include this line:

module.exports.initMaster = true;

To run a function on the master process include something like this:

module.exports.initMaster = function(defer, engine, log) {
    try {
        //Perform master logic... 
        defer.resolve();
    } catch(e) {
        defer.reject(e); //Oh no an error! This will shutdown the engine.
    }
};

Configuration

Configurations for KNE are defined per module/submodule. KNE uses these definitions to generate a configuration file configs/config.js in the root folder of your project on startup of the engine, if the file does not exist. KNE will also fill out each configuration with its default specified in its definition. If a configuration is not defined in configs/config.js and no default is set in its definition then KNE will refuse to start.

An example of a configuration definitions:

//Inside of module prj-my-module
module.exports.configs = {
    url: 'http://example.com/', //Define a configuration with a default.
    blah: 13421,
    test: null,
    secret: undefined //Define a configuration with out a default, this will be required to be filled in.
};

//Require some other modules/submodules, like kne-mongo...

The example above would generate a configuration file like this:

module.exports = {
    'kne': { //Default configuration included for the engine its self.
        'workers': 4,
        'logLevel': 'info'
    },
    'kne-mongo': { //The configuration for kne-mongo.
        'url': 'mongodb://localhost/kne-default'
    },
    'prj-my-module': { //The configuration you defined for your module.
        url: 'http://example.com/',
        blah: 13421,
        test: null,
        secret: undefined //If this is not filled out, KNE will refuse to start.
    }
};

Using a configuration:

MyModule.prototype.init = function(defer) {
    var self = this;
    
    //If prj-my-module.url is not defined and does not have a default, this will throw an error.
    var url = self.engine.getConfig('prj-my-module.url');

    console.log('URL configuration:', url);

    defer.resolve();
};

Logging

Logging for KNE is handled by Bunyan, a JSON logging library. All logs written through Bunyan are stored as JSON and can be viewed with the Bunyan CLI. More information on this is available in the Bunyan documentation.

In development mode, logs are printed to stdout and filtered by the Bunyan CLI. (Which is why you need it installed when working with development mode.) In production mode, logs are written to the folder logs/ in the root of your project. Submodule logs are written to engine.log, while module logs are written to log files named after their module name.

Each module and submodule is given an instance of a Bunyan logger to use for logging. It can be used like so:

MyModule.prototype.init = function(defer) {
    var self = this;
    //...
    
    try {
        //Do something error prone...
        
        self.log.info('It worked!', someObject);
    } catch (e) {
        self.log.error('Oh no there was an error! Continuing anyway...', e);
    }
    
    //...
};

Project Creation

Projects created with KNE are essentially collections of modules that are either project specific or generic. This section will cover project structure, but for creating modules you can view the modules section.

You can install KNE with NPM:

npm install kne --save

Projects should follow a structure similar to this:

my-project/
    configs/    - Automatically generated.
    logs/   - Automatically generated in production mode.
    node_modules/   - NPM modules, automatically generated.
    modules/    - A folder containing all of your project specific modules.
        prj-my-project/ - An example module.
    .gitignore  - Ignore file for Git. Should ignore configs, logs, and node_modules. An example below.
    app.js  - The entry point for your project, an example is below.
    package.json    - Can be generated with the command 'npm init'.
    readme.md   - A README file describing your project that links back to the Project Setup section in this documentation.

app.js example:

global.KNE_BASE_PATH = __dirname; //Set the base path in which KNE will generate the configuration and logs folder.

var engine = require('knetik-node-engine'); //Require KNE, must be installed with NPM.
    cluster = require('cluster'); //Require the cluster module, used to see if this is the master process.

//Remember that this file is loaded on the master and each worker.
engine.init([
    'kne-notification-service', //Require a module that was installed with NPM.
    'prj-my-module' //Require a project specific module.
]).then(function() {
    if (cluster.isMaster) { //Ensure this is the master process that is finished. It will only return when all workers are started.
        console.log('Engine initialized!'); //This promise returns when the engine has finished initialization.
    }
}, function(err) {
    console.error('Engine failed to initialize!', err); //Returned when the engine failed to initialize, nothing needs to be done as KNE will shutdown.
});

.gitignore example:

node_modules
configs/config.js
logs

package.json example:

{
  "name": "my-project",
  "version": "1.0.0",
  "description": "An example project built with KNE.",
  "main": "app.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Knetik",
  "license": "ISC",
  "dependencies": {
    "knetik-node-engine": "[email protected]:knetikmedia/knetik-node-engine.git#v1.4.8", //Require KNE v1.4.8.
    "kne-notification-service": "[email protected]:knetikmedia/kne-notification-service.git#v1.0", //Require the Notification Service v1.0.
    "lodash": "^3.10.1",
    "q": "^1.4.1"
  }
}

Once all of this is setup, you can follow the Project Setup section.

Tests

Install Mocha globally:

npm install -g jasmine

Run the tests:

npm test

If all goes well, the process will exit with code 0. If any errors occur the process will exit with a code other than 0.