godot2

A streaming real-time event processor based on Riemann written in Node.js

Godot2 is a major rewrite for Node 6 and Streams2/Streams3 syntax, similar to as suggested in nodejitsu/godot#64. The async patterns and inheritance are simplified with Iced CoffeeScript.

Many thanks and much credit to the original authors at Nodejitsu, @indexzero and @jcrugzz.

• Usage
• Events
• Reactors
  • Primitives
• Producers
• Tests

Usage

Here is a simple example of a Reactor server that will send an email to [email protected] if the Producer server for app.server fails to send a heartbeat after 60 seconds.

  var godot = require('godot');

  //
  // Reactor server which will email `[email protected]`
  // whenever any service matching /.*\/health\/heartbeat/
  // fails to check in after 60 seconds.
  //
  godot.createServer({
    //
    // Defaults to UDP
    //
    type: 'udp',
    reactors: [
      function (socket) {
        return socket
          .pipe(godot.where('service', '*/health/heartbeat'))
          .pipe(godot.expire(1000 * 60))
          .pipe(godot.email({ to: '[email protected]' }))
      }
    ]
  }).listen(1337);

  //
  // Producer client which sends events for the service
  // `app.server/health/heartbeat` every 15 seconds.
  //
  godot.createClient({
    //
    // Defaults to UDP
    //
    type: 'udp',
    producers: [
      godot.producer({
        host: 'app.server.com',
        service: 'app.server/health/heartbeat',
        ttl: 1000 * 15
      })
    ],
    //
    // Add Reconnect logic that uses `back`
    //
    reconnect: {
      retries: 2,
      minDelay: 100,
      maxDelay: 300
    }
  }).connect(1337);

Events

Similar to Riemann, events in godot are simply JSON sent over UDP or TCP. Each event has these optional fields:

  {
    host:         "A hostname, e.g. 'api1', 'foo.com'"
    service:      "e.g. 'API port 8000 reqs/sec'",
    state:        "Any string less than 255 bytes, e.g. 'ok', 'warning', 'critical'",
    time:         "The time of the event, in unix epoch seconds",
    description:  "Freeform text",
    tags:         "Freeform list of strings, e.g. ['rate', 'fooproduct', 'transient']",
    meta:         "Freeform set of key:value pairs e.g. { 'ewma': 12345 }",
    metric:       "A number associated with this event, e.g. the number of reqs/sec.",
    ttl:          "A floating-point time, in seconds, that this event is considered valid for."
  }

Reactors

Reactors in Godot are readable and writable Stream instances which consume Events and produce actions or aggregate data flow. In the example above you may see that when we define the array of reactors by wrapping it with a simple function. This function has a single argument that represents the data coming over the wire. This data can be piped to any godot stream or any Transform stream you find on NPM!

~~Note Reactors are currently still streams1 streams (so they do not handle backpressure) but this will begin to change in the near future for node 0.12.x. (Performance reasons)~~ I'm about halfway through this. -@doublerebel

Primitives

There are several core Reactor primitives available in godot which can be composed to create more complex behavior:

• .aggregate(): Aggregates metric property on events
• .change(key {from: x, to: y}): Emits events when the key is changed, accepts optional from and to options for more specific changes.
• .email(options): Sends an email to the specified options.
• .expire(ttl): Emits an event when no data is received after ttl milliseconds.
• .forward(options): Forwards all events to a remote server located at options.host and options.port.
• .sms(options): Sends an sms to the specified options.
• .where(key, value)|.where(filters): Filters events based on a single key:value pair or a set of key:value filters.
• .rollup(interval, limit)|.rollup(options): Rollup a limit amount of events to emit every interval. interval can also be a function to allow you to create varying intervals (see below).

Rollup

Here are two possible rollup examples:

var godot = require('godot');

//
// Rolls up 10,0000 events every 5 minute interval
// then sends them in an email
//
var rollup = function (socket) {
  return socket
    .pipe(godot.rollup(1000 * 60 * 5, 10000))
    .pipe(godot.email({ to: '[email protected]' }))
}

//
// Scaling Rollup, rolls up 10,000 events every 5min interval for 1 hour,
// then rolls up 10,000 events every 30mins and emails them out
//

var scalingRollup = function (socket) {
  return socket
    .pipe(godot.rollup(function (period) {
      if(period < 12) {
        return 1000 * 60 * 5;
      }
      return 1000 * 60 * 30;
    }, 10000))
    .pipe(godot.email({ to: '[email protected]' }))
}

Producers

Producers in Godot are readable Stream instances which produce Events. Events will be emitted by a given Producer every ttl milliseconds.

Tests

All tests are written in vows and can be run with npm:

  npm test

Copyright (C) 2012. Charlie Robbins, Jarrett Cruger, and the Contributors.

License: MIT

Sound Wave designed by Alessandro Suraci from the thenounproject.com