elastic-logger

Install

npm install @niccsj/elastic-logger --save

Features

• Parses standard node.js error logs into clear JSON format and ships the logs from to an Elastic Search cluster. Enables easy visualization and aggregation in Kibana without the need of Logstash or any other log formatting tool.
• Includes a middleware which captures and ships all the incoming api logs along with necessary details.
• Also, has the gives you an option to export outgoing api calls for tracking and logging.
• Provides two custom Error classes and an errorHandler.
• Automated index creation .via templates. ILM enabled by default for rollover. A default policy is configured acc to values specified in constants.js.
• Automated index pattern creation in Kibana. Refreshes index pattern on each restart.
• Can be used as a stand alone logging library.
• Fields available as of now:
  • ErrorLogger:
    • name
    • message
    • description
    • meta
    • status
    • scope
    • type
    • metadata
    • parsed
  • AccessLogger:
    • url
    • method
    • headers
    • remoteAddress
    • remoteFamily
    • statusCode
    • statusMessage
    • processingTime
  • ApiLogger
    • url
    • processingTime
    • statusCode
  • CommonFields:
    • microService
    • logType
    • logDate
    • logDateTime
    • @timestamp

Pre-Requisites

• An elasticsearch cluster.
• Minimum 1 master node. Can provide multiple hosts for fault-tolerance.
• Kibana for visualization and visibility.

Functions and Classes available for import

• initializeElasticLogger({ esConnObj, brand_name, cs_env, microServiceName, batchSize, timezone, ilmObject = {}, indexSettings = {} }) Used to initialize elasticsearch client and setup default values.
• exportAccessLogs({ microServiceName, brand_name, cs_env, batchSize, timezone = 'Asia/Calcutta' }) Used to enable access logs shipping.
• exportErrorLogs({ err, ship = true, log = true, self = false, timezone = 'Asia/Calcutta', scope = '@niccsj/elastic-logger', status = null, exporter = false, batchSize, brand_name, cs_env, microServiceName }) Used to enable error logs shipping.
• errorHandler({ err, microServiceName, brand_name, cs_env, batchSize, timezone = 'Asia/Calcutta', scope = 'global', status = null }) A common error handler function.
• elasticError({ name, message, type = 'nodejs', status }) An extended Error class used within the package, can be used in code for custom errors.
• dynamicError({ name, message, metadata, type = 'nodejs', status }) An even extended elasticError, can be used in code to add custom error fields to the error object.
• esClientObj The elastic search client created within the library can be used within the code to add any custom functionality. Follows the standard elastic search API implementation

Usage

.env Example

#======== elk config =======#

#micro service name for elk
MS_NAME="ocs"
elasticEnv="dev"

#enable/disable elastic-loggers
exportLogs=true
exportAccessLogs=true
outGoingApiLogger=true

elasticUrl='https://1.1.1.1:9200,https://2.2.2.2:9200'
kibanaUrl='https://abc.xyz.com/kibana'

##elasticAuth
elasticAuthType='api'
elasticApiKey=''

elasticAuthType='basic'
elasticUser=''
elasticPass=''

elasticAuthType='none'

##index and ilm settings //edit to overwrite, comment for defaults
##this should be a valid json string
elasticTimezone='Asia/Clacutta'
elasticBatchSize='200'
ilmObject='{"size":"2gb", "hotDuration":"2d", "warmAfter":"1h", "deleteAfter":"5d", "shrinkShards":1, "overwriteILM":false }'
indexSettings='{ "primaryShards":3, "replicaShards":1, "overwrite":false }'

#======== elk config =========#

Access/API Logs

Import the package at the very start of you application. Generally it's, app.js

/*
  app.js
*/
const { exportAccessLogs, initializeElasticLogger } = require('@niccsj/elastic-logger');

Then, initialize it as follows. You can either set the dynamic values from .env as above, or hard code it.

const elasticLoggerObj = {};
elasticLoggerObj.esConnObj = {
  url: process.env.elasticUrl, //comma separated url of the ES nodes. Ex: elasticUrl='https://0.0.0.0:9200,https://0.0.0.0:9200'
  authType: process.env.elasticAuthType, //allowed values: 'none', 'basic', 'api'
  auth: { //required if authType !=none
    apiKey: process.env.elasticApiKey, //required for api //base64 encoded id:apiKey from elasticsearch
    username: process.env.elasticUser, //required for basic
    password: process.env.elasticPass //required for basic
  }, //if not provide, defaults to the values defined in constants.js
  ssl: { rejectUnauthorized: false }, //required if https is being used
  sniff: false, //optional. Please read sniffing document before enabling this option.
  sniffInterval: false, //optional
  sniffOnFault: false //optional
};
// to specify default values
elasticLoggerObj.brand_name = process.env.BRAND_NAME;
elasticLoggerObj.microServiceName = process.env.MS_NAME;
elasticLoggerObj.cs_env = process.env.elasticEnv ? process.env.elasticEnv : process.env.CS_ENV;
elasticLoggerObj.timezone = process.env.elasticTimezone ? process.env.elasticTimezone : 'Asia/Calcutta';
elasticLoggerObj.batchSize = Number(process.env.elasticBatchSize ? process.env.elasticBatchSize : 100); //logs are shipped in batches to improve speed and reduce load on elastic stack. IMP: do not set too low value, when expecting high number of requests.

elasticLoggerObj.ilmObject = (process.env.ilmObject) ? JSON.parse(process.env.ilmObject) : { "size": "500mb", "hotDuration": "2d", "warmAfter": "1h", "deleteAfter": "5d", "shrinkShards": 1, "overwriteILM": false }; //explanation below
//  ilmObject: { //These objects can be omitted entirely if the default values from constants.js matches your need
//     size: '500mb', //max size before index rollover
//     hotDuration: '2d', //max time index stays in hot phase
//     warmAfter: '1h', //time to wait to move index to warm phase after rollover
//     deleteAfter: '5d', //index is deleted after
//     shrinkShards: 1, //shrinks the number of shards an index was in hot phase
//     overwriteILM: false //allows you to overwrite an existing ILM policy. Use with care
//   },
elasticLoggerObj.indexSettings = (process.env.indexSettings) ? JSON.parse(process.env.indexSettings) : { "primaryShards": 3, "replicaShards": 1, "priority": 1, "overwrite": false }; //explanation below
  // indexSettings: { //These objects can be omitted entirely if the default values from constants.js matches your need
  //   primaryShards: 3, //no of shards you want to allocate to an index. Can be more that nodes available.
  //   replicaShards: 1, //no of replicas to maintain. Should be at least 1 to avoid data loss
  //   overwrite: false, //allows you to overwrite an existing template. Use with care
  // }
};
elasticLoggerObj.exportApiLogs = (process.env.outGoingApiLogger) ? true : false; //enables or disables the ability to ship outbound api logs to es. Defaults to true
initializeElasticLogger(elasticLoggerObj);

Then, we need to pass the function to the very first express middleware after all servers settings have been defined.

if (process.env.exportAccessLogs) {
  const elasticLoggerObject = { //defaults from constants.js
    microServiceName: 'ocs',
    brand_name: process.env.BRAND_NAME,
    cs_env: process.env.CS_ENV,
    batchSize: 2,
    timezone: 'Asia/Calcutta',
  };
  app.use(exportAccessLogs({})); //pass an empty object, if values from constants.js matches your need.
  app.use(exportAccessLogs(elasticLoggerObject)); //invoke like this if you feel the need to override some settings like batchSize etc.
}

ErrorLogs

/*
  www
*/
const { exportErrorLogs, errorHandler } = require('@niccsj/elastic-logger');

To enable error logs capturing, we'll listed on error events and invoke our functions as follows:

process.on('unhandledRejection', (reason, promise) => {
  const errReason = reason.stack || reason;
  const err = `Reason--${errReason}`;
  if (process.env.elasticUrl) {
    const elasticLoggerObject = { //except err, all values are needed only if you plan to overwrite the values set during initialization.
      err: reason, //Required. You can also pass the variable err if you want less verbose error output.
      microServiceName: 'ocs',
      brand_name: process.env.BRAND_NAME,
      cs_env: process.env.CS_ENV,
      batchSize: 0,
      timezone: 'Asia/Calcutta',
      scope: 'unhandledRejection' //This can be any value as per your need.
    };
    exportErrorLogs(elasticLoggerObject);
  }
});

Similarly,

process.on('uncaughtException', (err, origin) => {
  const errReason = reason.stack || reason;
  const err = `Reason--${errReason}`;
  if (process.env.elasticUrl) {
    const elasticLoggerObject = { //except err, all values are needed only if you plan to overwrite the values set during initialization.
      err, //Required.
      scope: 'uncaughtException' //This can be any value as per your need. //can be omitted
    };
    exportErrorLogs(elasticLoggerObject);
  }
});

To collect error exceptions caught with try/catch, you can either your your own errorHandling function or use the one included in this package.

• Own Function:

  const handleAppError = async ({ err }) => {
    try {
      if (process.env.elasticUrl) {
        const elasticLoggerObject = { //except err, all values are needed only if you plan to overwrite the values set during initialization.
          err,
          microServiceName: 'ocs',
          brand_name: process.env.BRAND_NAME,
          cs_env: process.env.CS_ENV,
          batchSize: 0,
          timezone: 'Asia/Calcutta',
          scope: 'handleAppError'
        };
        exportErrorLogs(elasticLoggerObject);
      }
    } catch (err) {
      //
    }
  };
  
  const anyFunction = async (params) => {
    try {
      //operations
    } catch (err) {
       handleAppError({ err });
    }
  };
  

• IncludedFunction

  const anyFunction = async (params) => {
    try {
      //opration
      //add the fields to metadata which you need to track
      const metadata = {}; //object
      metadata.sessionId = ''; //string
      metadata.psid = ''; //string
      metadata.platform = ''; //string
      const status = 872 //number
  
    } catch (err) {
       errorHandler({ err, ship: false, metadata: metadata, scope: '@niccsj/elastic-logger.initializeElasticLogger' });
    }
  };

  • Arguments explanation for errorHandler:
    • err = err | Object The error object Required
    • ship = false/true | Bool Boolean to specify whether or not to ship this exception to elasticsearch. Can be useful if you need to silent some errors. Optional | Defaults to false
    • log = false/true | Bool Boolean to specify whether or not to log the error to the console .via console.error() Optional | Defaults to true
    • timezone = 'Asia/Calcutta' | String The timezone to be used. Optional | Defaults to 'Asia/Calcutta'
    • metadata = { anyObject } | Object Addition variables/fields can to added to make pin point the origin of error
    • scope = 'anyFunction' | String A string value that can define a score of the error. Can be anything, for example the name of the function where the error originated. Can be used to track errors by function. Optional | Defaults to '@niccsj/elastic-logger'
    • status = 00 | Number A custom error code for your errors, can be used to track similarly api errors Optional | Defaults to null
    • batchSize = 10 | Number Optional | Defaults to the values set during initialization
    • brand_name = 'brand' | String Optional | Defaults to the values set during initialization
    • cs_env = 'prod' | String Optional | Defaults to the values set during initialization
    • microServiceName = 'authService' | String Optional | Defaults to the values set during initialization

Error Class

The only difference between elasticError and dynamicError is the additional filed metadata in the latter.

const { dynamicError, elasticError } = require('@niccsj/elastic-logger');
const anyFunction = async () => {
  try {
    //some condition

    //use of metadata same as explained above
    //use case if you specifically want to throw an error based on a condition with specific values for identification.

    throw new dynamicError({ name: `${someName}`, message: `some message ${variable}`, metadata, status: `${status}`, type: 'optional' });

  } catch (err) {
    //handle error
  }
};