easy-log-watcher

A Log Watcher Utility plugin.

Last update: 07/11/2018

Description

This plugin provides monitoring of logs and runs triggers depending on the data found in the logs. The monitoring and triggering mode are fully user-defined.

Why this plugin

An application can generate logs. Some log lines can come from other plugins and can not be monitored in real time by the application itself. The application should periodically monitor the logs and act according to the information read. This is the purpose of this plugin.

Example

Application A uses the B plugin. Sometimes plugin B crashes, and generates this type of log:

{"error":"plugin B cannot respond","level":"error","when":1529559892669}

By running periodically, the log watcher will read the line from the A log and trigger the following action:

restart B

How it works

Plugin declaration

Your application must declare this plugin:

const logWatcher = require('easy-log-watcher')

start

To start the log watcher, simply call the start function, which is a Promise running in asynchronous mode:

logWatcher.start(analyzers, options)

• analyzers: an array of analyzer objects. If null or empty, the log watcher stops immediatly. Otherwise the log watcher will run by looping periodically on all these analyzers.
• options: an object. See the log watcher options below.

This start() promise returns { success: true }.

## stop

To stop the current log watcher, simply call the stop function:

logWatcher.stop()

This stop() function returns false.

Note: the log watcher can also be stopped during a parser process. See the Parser with log watcher stop Example below.

Examples

Running in background:

const logWatcher = require('easy-log-watcher')
// Starts the log watcher in asynchronous mode
logWatcher.start([ /* your analyzers here */ ])
// your code here while the log watcher is running
...

Full promise code:

const logWatcher = require('easy-log-watcher')
// Starts the log watcher in asynchronous mode
logWatcher.start([ /* your analyzers here */ ])
.then(function (result) {
  // your code here when the log watcher has finished
  ...
})
.catch(function (err) {
  // your code managing the start error
  ...
})
// your code here while the log watcher is running
...

Start and stop:

const logWatcher = require('easy-log-watcher')
// Starts the log watcher in asynchronous mode
logWatcher.start([ /* your analyzers here */ ])
// your code here while the log watcher is running
...
// It's time to stop the log watcher
logWatcher.stop()

Log Watcher Options

The Log Watcher options pattern is:

  {
    delay: n, // an integer
    logger: { /* an object or null */ },
    sequential: aBoolean // true or false
  }

• delay: an integer, default = 3600 (1 hour). The time in seconds between two loops.
• logger: an object, default = console. The logger to output the info messages from the log watcher. If you don't want messages from the log watcher, please set this value to null.
• sequential: a boolean, default = false. The running mode of the asynchronous analyzers. If true, the analyzers will be processed sequentially. Otherwise the analyzers will be processed in parallel mode, which is more efficient.

Example

logWatcher.start(
  [ /* your analyzers here */ ],
  {
    delay: 60, // 1 minute
    logger: null, // no info messages
    sequential: true // the analyzers will be processed in sequence
  }
)

Delay

The log watcher will periodically analyze the log files by looping on the analyzers. The delay between two loops is set in the options. Its value is in seconds. During one loop, the log watcher runs all the analyzers. For further explanation of a loop, see the Process in detail chapter below.

Analyzer

The log watcher can use several analyzers. One analyzer defines the logs files to read and the parsers to run on them. The analyzer pattern is:

{
  filename: 'a file name pattern',
  parsers: [ /* a list of parsers */ ]
  options: { /* this analyzer options */ }
}

• filename: a string. The name of the log files to be monitored. If null or empty, the analyzer do nothing. This string can be a pattern.

  Note: this log watcher uses the line-by-line plugin (thanks to that!), which is efficient to read big files without buffering the files into memory.

• parsers: an array of parser objects. If null or empty, the analyzer do nothing.

• options: an object. See the analyzer options below.

Analyzer options

Your application can specify how the files will be read and how the parsers will be executed. The Analyzer options pattern is:

{
  sequentialFiles: aBoolean, // true or false
  sequentialParsers: aBoolean // true or false
}

• sequentialFiles: a boolean, default: false. If true, the log files are read sequentially. Otherwise, they are read in parallel mode, which is more efficient.

• sequentialParsers: a boolean, default: false. If true, the parsers are executed sequentially. Otherwise, they are executed in parallel mode, which is more efficient.

Analyzer simple example

{
  filename: '/home/myuser/logs/app-001.log',
  parsers: [{
    checker: function (line, values) {
      if (line.indexOf('error') > -1) { return true }
      else { return false }
    },
    trigger: function (file, line, values) {
      console.log('line with error', line)
      return 0
    }  
  }]
}

This analyzer reads the log /home/myuser/logs/app-001.log line by line.

If a log line contains the text error, the trigger is executed: it writes a message on the console.

This analyzer will continue until all lines of the log are read.

File name pattern

The file name is a string that can use some patterns, like:

/home/myuser/logs/app*.log

By example, these files will be processed by the log watcher:

/home/myuser/logs/app-website-01.log
/home/myuser/logs/app-website-02.log
/home/myuser/logs/app.1-2-3.txt.log
/home/myuser/logs/app.log

These files will not be processed by the log watcher:

/home/myuser/logs/another-app.log
/home/myuser/logs/app-website-01.log.txt

This log watcher uses the fast-glob plugin (thanks to that!) to retrieve the files from the patterns. For more informations on the patterns, see the fast-glob project.

Parser

The analyzer can use several parsers to examine the log lines and run some triggers. The parser pattern is:

  {
    checker: function (line, values) { /* your checker function here */ },
    trigger: function (file, line, values) { /* your trigger function here */ },
    values: { /* your parser values */ }
  }

• checker: a user-defined function that checks whether the line contains values.

This function must accept 2 arguments:

1. line: a string. The log line string to process.

2. values: an object. The values argument of this parser.

This function must return a boolean. If true, the trigger function of this parser is called. Otherwise, the parser continue with the following log lines.

• trigger: a user-defined function that executes a process. The process can be of any kind, we know that your imagination has no limit :)

This function must accept 3 arguments:

1. file: a string. The current name of the log read by the analyzer.
2. line: a string. The current log line analyzed by the checker.
3. values: an object. The values argument of this parser.

This function must return an integer: the trigger exit code. Depending on this integer, the parser, the analyzer or the log watcher can be stopped immediatly. Returning the 0 value, the parser continue with the following log lines. See the trigger exit code chapter for more explanation.

• values: an optional object that is passed as argument to the checker and the trigger. This is useful if your checker or trigger function needs data from your application.

Parser simple example

{
  checker: function (line, values) {
    if (line.indexOf('error') > -1) { return true }
    else { return false }
  },
  trigger: function (file, line, values) {
    console.log(values.foo, 'file', file, 'line with error', line)
    return 0
  },
  values: { foo: 'bar' }
}

Only a line containing the text error will start the trigger.

The trigger will write a message on the console. The message contains some data from the values argument.

The parser will continue because the trigger returns 0. Thus all lines of the file are analyzed.

Parser with log watcher stop Example

Your application can stop the log watcher during the parser process:

{
  checker: function (line, values) {
    if (line.indexOf('error') > -1) { return true }
    else { return false }
  },
  trigger: function (file, line, values) {
    console.log(values.foo, 'file', file, 'line with error', line)
    return 5 /* The log watcher will stop */
  },
  values: { foo: 'bar' }
}

Only a line containing the text error will start the trigger.

The trigger will write a message on the console. The message contains some data from the values argument.

The parser will stop and the log watcher will stop because the trigger returns 5.

Trigger exit code

The trigger function must return an integer: its exit code. Depending on this integer, the process can be continued or stopped:

| Code | Action | | ----- | ---------------------------------------- | | 0 | The parser will read the next line. | | 1 | The parser stops. The analyzer runs the next parser. | | 2 | All the analyzer parsers are stopped. The analyzer will examine the next log file. | | 3 | The analyzer stops. The log watcher will start the next analyzer. | | 4 | All the analyzers are stopped. The log watcher runs a new loop. | | 5 | The log watcher stops. All the processes are canceled. | | other | The parser will read the next line. |

The process in detail

1. The log watcher analyzers are processed in parallel or sequential mode.
2. One analyzer is used. It finds the log files to read in parallel or sequential mode.
3. One log file is found. The analyzer reads and process Its lines one by one sequentially.
4. One log line is read. All the analyzer parsers process this line in parallel or sequential mode.
5. The current log line is used by one parser. If the parser checker returns true, the parser trigger is executed.
6. If the parser trigger returns 1, the parser stops. Parallel mode: the analyzer reads the next line > step 4. Sequential mode: the analyzer runs the next parser on the same line > step 5.
7. If the parser trigger returns 2, all the analyzer parsers are stopped. The analyzer reads the next line > step 4.
8. If the parser trigger returns 3, the analyzer stops. Parallel mode: the log watcher waits during the delay set in the options, then restarts the process > step 1. Sequential mode: the log watcher starts the next analyzer > step 2.
9. If the parser trigger returns 4, all the analyzers are stopped. The log watcher waits during the delay set in the options, then restarts the process > step 1.
10. If the parser trigger returns 5, the log watcher stops immedialty.
11. If the parser trigger returns another value (as 0), the parser uses the next log line > step 5.
12. If there is no more line, the next parser is started (sequential mode > step 4).
13. If there is no more parser, the next log file is read (sequential mode > step 3).
14. If there is no more log file, the log watcher uses the next analyzer (sequential mode > step 2).
15. If there is no more analyzer, the log watcher waits during the delay set in the options, then restarts the process > step 1.

Reading big files

This log watcher uses the line-by-line plugin (thanks to that!) which is efficient to read big files without buffering the files into memory.

Install

To install, simply use npm:

npm install easy-log-watcher

Test

To run tests, simply use npm:

npm test

Contributing

If you feel you can help in any way, be it with documentation, examples, extra testing, or new features please get in touch.

License

Copyright (c) 2018 e-soa Jacques Desodt. Licensed under MIT.