monitor: a service monitor for node.js

What is monitor?

• watchmen monitors health (outages, uptime, response time warnings, avg. response time, etc) for your servers.
• ping types are pluggable through npm modules. At this time, http-head and http-contains are available. Read more about ping services and how to create one below.
• watchmen provides custom actions through plugins (console outpug, email notifications, etc).
• the code base aims to be simple and easy to understand and modify.

Installation

Requirements

Get redis from redis.io and install it.

Installing watchmen

Clone the repo by using

$ git clone [email protected]:SamsclubdotcomUS/watchmen.git

or

$git clone https://github.com:SamsclubdotcomUS/watchmen.git

Then install the required dependencies using npm

$ cd watchmen
$ npm install

Running and stopping watchmen

Make sure you have redis-server in your PATH. Then you can run watchmen services:

$ redis-server redis.conf
$ node run-monitor-server.js
$ node run-web-server.js

Development workflow

Make sure bower is installed globally:

$ npm install -g bower

Fetching bower dependencies

$ bower install

Re-building watchmen assets

$ gulp build

Dev watch

$ gulp watch

Running tests

See below.

Managing your node processes with pm2

Install pm2:

$ npm install -g pm2

Configure env variables:

$ export WATCHMEN_WEB_PORT=8080

Run servers:

$ pm2 start run-monitor-server.js
$ pm2 start run-web-server.js

Server list:

$ pm2 list

Managing processes with node-foreman

node-foreman can be used to run the monitor and web server as an Upstart service. On Ubuntu systems, this allows the usage of service watchmen start.

Watchmen already include a Procfile so you can also manage with nf.

$ npm install -g foreman
$ nf start

To export as an Upstart script using the environment variables in a .env file:

$ PATH="/home/user/.nvm/versions/v5.1.0/bin:$PATH" nf export -o /etc/init -a watchmen

You can run this without the -o /etc/init flag and move the files to this directory (or the appropriate Upstart) directory yourself. Make sure you have the correct path to the node bin, you can find out with which node.

More documentation on node-foreman:

https://github.com/strongloop/node-foreman

Configuration

Config is set through env variables.

Have a look at the /config folder for more details, but the general parameters are:

export WATCHMEN_BASE_URL='http://localhost'
export WATCHMEN_WEB_PORT='8080'

Ping services

Embedded ping services

HTTP-HEAD

https://www.npmjs.com/package/watchmen-ping-http-head

HTTP-CONTAINS

https://www.npmjs.com/package/watchmen-ping-http-contains

Creating your own ping service

Ping services are npm modules with the 'watchmen-ping' prefix.

For example, if you want to create a smtp ping service:

a) create a watchmen-ping-smtp module and publish it. This is how a simple HTTP ping service looks like:

var request = require('request');

function PingService(){}

exports = module.exports = PingService;

PingService.prototype.ping = function(service, callback){
  var startTime = +new Date();
  request.get({ method: 'HEAD', uri: service.url }, function(error, response, body){
    callback(error, body, response, +new Date() - startTime);
  });
};

PingService.prototype.getDefaultOptions = function(){
  return {}; // there is not need for UI confi options for this ping service
}

b) npm install it in watchmen:

     npm install watchmen-ping-smtp

c) create a service that uses that ping service

Nodemailer Notifications plugin (third party contribution)

https://www.npmjs.com/package/watchmen-plugin-nodemailer

Slack Notifications plugin (third party contribution)

https://www.npmjs.com/package/watchmen-plugin-slack

Creating your own custom plugin

A watchmen instance will be injected through your plugin constructor. Then you can subscribe to the desired events. Best is to show it through an example.

This what the console plugin looks like:

var colors = require('colors');
var moment = require('moment');

var eventHandlers = {

  /**
   * On a new outage
   * @param {Object} service
   * @param {Object} outage
   * @param {Object} outage.error check error
   * @param {number} outage.timestamp outage timestamp
   */

  onNewOutage: function (service, outage) {
    var errorMsg = service.name + ' down!'.red + '. Error: ' + JSON.stringify(outage.error).red;
    console.log(errorMsg);
  },

  /**
   * Failed ping on an existing outage
   * @param {Object} service
   * @param {Object} outage
   * @param {Object} outage.error check error
   * @param {number} outage.timestamp outage timestamp
   */

  onCurrentOutage: function (service, outage) {
    var errorMsg = service.name + ' is still down!'.red + '. Error: ' + JSON.stringify(outage.error).red;
    console.log(errorMsg);
  },

  /**
   * Failed check (it will be an outage or not according to service.failuresToBeOutage
   * @param {Object} service
   * @param {Object} data
   * @param {Object} data.error check error
   * @param {number} data.currentFailureCount number of consecutive check failures
   */

  onFailedCheck: function (service, data) {
    var errorMsg = service.name + ' check failed!'.red + '. Error: ' + JSON.stringify(data.error).red;
    console.log(errorMsg);
  },

  /**
   * Warning alert
   * @param {Object} service
   * @param {Object} data
   * @param {number} data.elapsedTime (ms)
   */

  onLatencyWarning: function (service, data) {
    var msg = service.name + ' latency warning'.yellow + '. Took: ' + (data.elapsedTime + ' ms.').yellow;
    console.log(msg);
  },

  /**
   * Service is back online
   * @param {Object} service
   * @param {Object} lastOutage
   * @param {Object} lastOutage.error
   * @param {number} lastOutage.timestamp (ms)
   */

  onServiceBack: function (service, lastOutage) {
    var duration = moment.duration(+new Date() - lastOutage.timestamp, 'seconds');
    console.log(service.name.white + ' is back'.green + '. Down for '.gray + duration.humanize().white);
  },

  /**
   * Service is responding correctly
   * @param {Object} service
   * @param {Object} data
   * @param {number} data.elapsedTime (ms)
   */

  onServiceOk: function (service, data) {
    var serviceOkMsg = service.name + ' responded ' + 'OK!'.green;
    var responseTimeMsg = data.elapsedTime + ' ms.';
    console.log(serviceOkMsg, responseTimeMsg.gray);
  }
};

function ConsolePlugin(watchmen) {
  watchmen.on('new-outage', eventHandlers.onNewOutage);
  watchmen.on('current-outage', eventHandlers.onCurrentOutage);
  watchmen.on('service-error', eventHandlers.onFailedCheck);

  watchmen.on('latency-warning', eventHandlers.onLatencyWarning);
  watchmen.on('service-back', eventHandlers.onServiceBack);
  watchmen.on('service-ok', eventHandlers.onServiceOk);
}

exports = module.exports = ConsolePlugin;

Storage providers

Redis

Data schema

service - set with service id's
service:latestOutages - latest outages for all services
service:<serviceId> - hashMap with service details
service:<serviceId>:outages:current - current outage for a service (if any)
service:<serviceId>:outages - sorted set with outages info
service:<serviceId>:latency - sorted set with latency info
service:<serviceId>:failurecount - number of consecutive pings failures (to determine if it is an outage)

Using fake data for development

cd scripts
sh populate-dummy-data-120days.sh # will populate data for a 120 day period

or

sh populate-dummy-data-30days.sh

etc..

Tests

$ npm test

Test coverage

$ npm run coverage

Then check the coverage reports:

$ open coverage/lcov-report/lib/index.html

Debugging

watchmen uses debug

set DEBUG=*

Contributing

You can contribute by:

• Addressing one if the items on the TODO list or one of the open issues.
• Creating monitor plugins.
• Creating ping services.
• Reporting bugs.