HashMonitor: Turn Logs Into Metrics Like A Boss

HashMonitor is a simple way to turn your logs into metrics by using hashtags in your log messages.

Log like this...

log("something #weird happened")
log("something #weird with #networking took #load_seconds=3")
log("something #weird with #networking took #load_seconds=2")

You'll get metrics like this...

{
    weird: { count: 3 },
    networking: { count: 2 },
    seconds: { count: 2, mean: 2.5, median: 3, ... },
}

How does it work?

By default, HashMonitor reads text lines from stdin, and every 30 seconds writes JSON statistics to stdout. You can pipe this JSON output into:

• your favorite stats dashboard (tinyfeedback, statsd, graphite, etc)
• your favorite alerting tool (pagerduty, nagios, etc)

Example: counting warning events

For example, you could keep track of warnings in your application by simply logging with a #warn hashtag:

log("something bad happened #warn")
log("something odd happened #warn")
log("something strange happened #warn")

HashMonitor will read your log file line-by-line, and count the number of #warn events:

{ warn: { count: 3 } }

Example: watching performance metrics

You can also track value-based metrics by assigning a numeric value to your hashtags:

log("speed was pretty fast #loaded_in_seconds=3")
log("speed was pretty fast #loaded_in_seconds=2")
log("something was slowwww #loaded_in_seconds=17")

HashMonitor will read your log file line-by-line, and count the number of #loaded_in_seconds events as well as other statistics:

{ loaded_in_seconds: 
   { count: 3,
     mean: 7.333333333333333,
     stddev: 6.847546194724712,
     x01: 17,
     x10: 17,
     median: 2,
     x90: 3,
     x99: 3,
     min: 17,
     max: 3 } }

How can I try it out?

Easiest is installing hashmonitor globally:

$ sudo npm install -g hashmonitor

Then just run the HashMonitor using the default stdin/stdout:

$ hashmonitor

Once you are here, just type some fake log messages into stdin:

hello #world
goodnight #moon
over the #moon

After 30 seconds you will see JSON stats output, like:

{ world: { count: 1 }, moon: { count: 2 } }

The counts are reset with each stats output. You can also try out value-based metrics:

slow stuff #loadtime=25
fast stuff #loadtime=10

Instead of just seeing counts now, you will also see additional stats about the distribution of these values:

{ loadtime: 
   { count: 2,
     mean: 17.5,
     stddev: 7.5,
     x01: 10,
     x10: 10,
     median: 25,
     x90: 25,
     x99: 25,
     min: 10,
     max: 25 } }

There ya go Boss!

How can I use HashMonitor to monitor my logs in {my favorite language}?

The easiest way to pipe logfile over stdin into HashMonitor. Simple example:

tail -F /var/log/my-service.log | hashmonitor

...this will output JSON statistics about your hashtags every 30 seconds.

How can I use HashMonitor to monitor browser-side Javascript?

HashMonitor has built-in HTTP access log parsing. You can slam logs into your favorite server (nginx, lighty, express, etc) with some simple Javascript:

function myLogger(message) {
  (new Image).src =
    '//log.example.com/?hashmonitor=' +
    encodeURIComponent(message)
}

...hashmonitor will parse the "hashmonitor" query argument automatically:

tail -F access.log | hashmonitor --parse-http-access

...and you'll still see JSON statistics output for your hashtags every 30 seconds :)

References

• original blog post
• JSConf slides