Chronica

A Node daemon to monitor your JSON services and web sites - their status and content.

For example, at a specified interval we send an HTTP HEAD request, and check that the status code is 200 (or 403, if so specified).

Furthermore, we can check the content for correctness - JSON or HTML.

Alerts are sent to specified email addresses and/or a Slack channel via your Slackbot.

A single YAML configuration file is used for all the components of the app, e.g.:

urlMonitor:
  interval: 45000 # check status every 45 seconds
  services:
    service1: https://myserver.com/service1

Pros

• YAML configuration
• Slack integration
• built for Node using ES6/7 via Babel
• modular design, so readily extensible
• JSON content monitoring e.g. expected properties and their types
• HTML content monitoring e.g. page title, canonical link etc.

Cons

• too immature for a stable release
• no database, history or fancy graphs
• requires restart for config changes to take effect
• incomplete, e.g. HTTP POST checks not yet implemented

HTML monitoring

Example configuration for the HTML monitoring component:

htmlMonitor:
   scheduledInterval: 45000 # check every 45 seconds
   services:
      mynews-facebook-sharing:
         url: http://mynews.com/my-test-article
         headers:
            User-Agent: facebookexternalhit # test facebook sharing
         content:
            title: "My test article about nothing"

where we wish to test Facebook sharing and so specify facebookexternalhit as the user-agent, via the ExpressJS headers for the HTTP request.

For HTML monitoring, we plan to add tests for:

• elements by CSS selectors, e.g. using cheerio

The HtmlMonitor component has defined defaults for regex for the content values. In particular, see our HtmlMonitor.yaml class decorator:

class: HtmlMonitor
source: HtmlMonitor.js
assign: [regex] # merge properties from defaults.regex into config.regex
defaults:
  timeout: 4000 # for HTTP request
  scheduledInterval: 45000 # check every 45 seconds
  regex:
     title: <title>(.+)</title>
     canonicalLink: <link rel="canonical" href="([^"]+)"/>

where we define the default regex for our page title and canonical link, and will Object.assign any additional regex properties from .chronica.yaml.

https://github.com/evanx/chronica/blob/master/components/HtmlMonitor.yaml

Custom regex

You can add custom regex in your .chronica.yaml as follows:

htmlMonitor:
   class: HtmlMonitor
   loggerLevel: debug
   regex:
     description: <meta name="description" content="(.*)">

Then description can then be used for a content check e.g.:

   services:
      beta:
         url: http://beta.iol.co.za
         content:
            description: Independent Online

JSON monitoring

Example configuration for the JSON monitoring component:

jsonMonitor: # name of the component instance
   scheduledInterval: 45000 # check every 45 seconds
   services:
      hn-api:
         url: https://hacker-news.firebaseio.com/v0/item/160705.json
         required: # ensure that the JSON response has the following props/types
            id: string
            time: integer
            type: string

Installing

git clone https://github.com/evanx/chronica &&
  cd chronica &&
  cat package.json &&
  npm install &&
  git submodule init &&
  git submodule update

Note we have a submodule dependency on https://github.com/evanx/redexutil for generic utils for ES7. (We use ES7 async functions, via Babel.)

Our scripts use pm2 for process management, and bunyan for showing logs. In order to use these scripts, you should install bunyan and pm2 globally:

 sudo npm install bunyan pm2 -g

git submodules

The util/ and scripts/ directories are git submodules.

See .gitmodules and .git/config

evans@boromir:~/chronica$ cat .gitmodules
[submodule "util"]
	path = util
	url = https://github.com/evanx/redexutil
[submodule "scripts"]
	path = scripts
	url = https://github.com/evanx/chronica-scripts

These might be, but should not be, [email protected] URLs e.g. [email protected]:evanx/redexutil.git

That can be fixed as follows.

cd ~/chronica/util
git remote set-url origin https://github.com/evanx/redexutil
git remote -v

cd ~/chronica/scripts
git remote set-url origin https://github.com/evanx/chronica-scripts
git remote -v

cd ~/chronica
git submodule sync
cat .gitmodules
cat .git/config | grep redexutil
cat .git/config | grep chronica-scripts

Alternatively, create them from scratch as follows:

git submodule deinit scripts
git submodule deinit util
git submodule add https://github.com/evanx/redexutil util
git submodule add https://github.com/evanx/chronica-scripts scripts

Sample config file

loggerLevel: info
alerter:
   elapsedThreshold: 600000 # suppress alerts for 10 minutes (self or peer)
   peers: # suppress alert if any peer alerted in past 10 minutes
      mypeer1:
         url: http://chronica.mypeer1.com/chronica
emailMessenger:
   fromEmail: [email protected]
   admins: # email recipients for alerts
   - email: [email protected]
   - email: [email protected]
slackMessenger:
   bots: # see: https://api.slack.com/slackbot
   - url: https://MY.slack.com/services/hooks/slackbot?token=...
reporter:
   helloDelay: 16000 # send an email 16 seconds after starting
   daily: # send a daily digest of current status of all services
      hour: 16 # 16:10 (pm)
      minute: 10
tracker: # tracks the status and decides if and when the send an email alert
  debounceCount: 2 # status must stay changed during multiple iterations before alert

The debounce count is important for debouncing flaky services where we expect suprious errors, and only want to be alerted when the service appears to go down and stay down (or stay up).

We have implemented three monitoring components, namely:

• URL monitoring
• HTML content monitoring e.g. the <title> element or some other regex
• JSON content monitoring e.g. expected properties and their types

URL HTTP status monitor

urlMonitor:
  interval: 45000 # check status every 45 seconds
  timeout: 8000 # HTTP connection timeout after 8 seconds
  services:
    hn: https://news.ycombinator.com
    myserver1: http://myserver1.com
    myserver2: http://myserver2.com

where the HTML title check is equivalent to the following curl command:

curl -sIL https://news.ycombinator.com | grep '^HTTP'

HTML content monitor

htmlMonitor:
   class: HtmlMonitor
   loggerLevel: debug
   scheduledTimeout: 2000 # initial check after 2 seconds
   scheduledInterval: 45000 # check every 45 seconds
   services:
      google:
         url: http://www.google.com
         content:
            title: "Google"

where the HTML title check is equivalent to the following curl command:

curl -sL google.com | grep '<title>' | head -1 | sed 's/.*<title>\([^<]*\).*/\1/'

JSON endpoint monitor

jsonMonitor:
   class: JsonMonitor
   loggerLevel: debug
   scheduledTimeout: 2000 # initial check after 2 seconds
   scheduledInterval: 45000 # check every 45 seconds
   services:
      hn-api:
         url: https://hacker-news.firebaseio.com/v0/item/160705.json?print=pretty
         required:
            id: string
            time: integer
            type: string

See https://github.com/evanx/chronica/blob/master/sample-config.yaml

Running

You must create your own configuration file e.g. ~/.chronica.yaml.

The scripts/ directory is a submodule, namely https://github.com/evanx/chronica-scripts

These scripts assist with the author's workflow. Consider forking this, and modifying as suits.

The scripts require:

• current working directory is chronica/ i.e. from git clone
• ~/.chronica.yaml exists, e.g. see sample/sample-config.yaml

See scripts/run.sh

  node index.js ~/.chronica.yaml debug | bunyan -o short

where we specify the config file.

Also see scripts/restart.pm2.sh which includes the following command:

cd ~/chronica
pm2 start index.js --name chronica -- ~/.chronica.yaml

You can tail -f the log file as follows:

ls --sort=time ~/.pm2/logs/chronica-out-*.log |
    head -1 | xargs tail -f | bunyan -o short

Recommended deployment configuration

Note that if you use multiple instances with the same config file i.e. monitoring the same endpoints, you can expect duplicate alerts by default. Perhaps use one instance to monitor all your endpoints, and another remote instance to monitor the monitor ;)

In order to monitor Chronica remotely, include the following in its config file:

expressServer:
   location: /chronica
   port: 8882

and proxy as required via NGINX or other.

In the systemReporter above, the disk is a percentage value, load is the current load average, and the redis is the current Redis memory usage in megs.

Alerting peers

It is possible to deploy multiple instances monitoring the same endpoints, and still avoid duplicate alerts via the following configuration:

alerter:
  elapsedThreshold: 300000
  peers:
    server1: http://chronica.server1.com/chronica

In this case, Chronica will check its peers before sending an alert. If any of its peers have sent any alerts in the past 5 minutes, then it will suppress the alert.

The elapsedThreshold is used for both itself and its remote peers, to suppress alerts for the configured duration after an alert is sent.

Other documentation

Implementation overview: https://github.com/evanx/chronica/blob/master/lib/readme.md

Component factory overview: https://github.com/evanx/chronica/blob/master/lib/ComponentFactory.md

Future work

We may copy these components into Redex, our modular/CSP project.

See: https://github.com/evanx/redex

Other resources

Redex utils: https://github.com/evanx/redexutil

Wiki home: https://github.com/evanx/vellum/wiki