Madgex Datadog Logging and Monitoring

All your Hapi + Datadog needs, in one handy package.

Usage

As a library

When used as a module, this library exports:

• hapi-pino
• autoLogErrors, a Hapi plugin to automatically log responses over a certain status code threshold, and all responses if passed a debug log level
• a function to set up dd-trace

There are no mutual dependencies so each can be used independently of the others. You can use them in the setup file for your server like so:

const { pino, autoLogErrors, trace, dataDogStats } = require('@madgex/datadog-monitoring');

async function createServer() {
  await trace({
    hostname: DD_AGENT_HOSTNAME || '',
    service: 'my-service-name',
    hapiOptions: {
      blacklist: ['/healthcheck']
    },
  });

  const Hapi = require('hapi');

  const server = new Hapi.Server({
    //etc
  })

  await server.register([
    {
      plugin: autoLogErrors,
      options: {
        level: LOG_LEVEL
      }
    },
    {
      plugin: pino,
      options: {
        // prettyPrint has been deprecated in pino
        // if you want to use it, you can add 'pino-pretty' as a dev dependency to your service
        // and implement it as below (see options here: https://github.com/pinojs/pino-pretty)
        // (note: pino-pretty should only be used in development)
        ...(IS_DEV && {
          transport: {
            target: 'pino-pretty',
          },
        }),
        level: LOG_LEVEL,
        redact: ['req.headers.authorization'],
        ignorePaths: ['/healthcheck'],
      },
    },
    {
      plugin: dataDogStats,
      options: {
        DD_AGENT_HOSTNAME,
        DD_AGENT_DSTATS_PORT,
      },
    },
  ]);

  return server;
}

Note that the tracer must be initialised before requiring Hapi, in order to correctly initialise the APM.

All available options for the dd-trace Hapi plugin can be passed as hapiOptions. hostname, if not set, will default to the discoverable Datadog agent host on AWS. The trace function returns the tracer instance so further plugin configuration can be added if you wish, eg:

async function createServer() {
  const tracer = await trace({
    hostname: DD_AGENT_HOSTNAME || '',
    service: 'my-service-name',
    debug: true // enables debugging the tracer, do not enable in production
    version: pkg.version,
    profiling: true,
    analytics: true,
    hapiOptions: {
      blacklist: ['/healthcheck']
    },
  });

  tracer.use('redis', { analytics: true });

  // etc
}

The hapi-pino plugin should be set up as described in its documentation.

The autoLogErrors plugin accepts two config options:

• level: the application's log level, to determine whether to log all requests. Defaults to 'info'.
• threshold: the status code above which responses should be logged as 'warn'. Defaults to 399.

The dataDogStats plugin decorates your hapi server object with dStats client. This client can call dstats functions like increment() with parameters:

  server.dataDogStats.increment(name, value, tags)

• name: Required - string, containing name of the graph e.g. "jobseeker_frontend_jobdetails"
• value: Required - number of increments e.g. 1
• tags: Required - array of strings, containing your values e.g. [similarJobsCallSuccess:success]

This plugin is running hot-shots module. Full usage docs here: https://www.npmjs.com/package/hot-shots#check

From the command line

This library also includes a custom transport to pipe Pino logs from a server's stdout to a Datadog agent over UDP, transforming the JSON format for processing and display. It works by running the server in a separate, child process and piping that process's stdout stream through a transform and write stream to send to the Datadog agent. It's intended to be used as a replacement for the usual node /entry/point.js in the npm start script, like so:

"start": "dd-monitor /path/to/server.js --hostname [hostname] --port [port] --echo --debug"

It accepts the following flags:

• hostname/h: the hostname of the Datadog agent. Defaults first to a DD_AGENT_HOSTNAME environment variable, and then to looking up the discoverable host on AWS
• port/p: the port to log to. Will default to a Logging__DataDogLoggingPort environment variable, but is otherwise required
• echo/e: echoes all logs to stdout, including a warning when a log was not parseable. Do not enable in production
• debug/d: log to stdout when a packet is transmitted to the UDP socket. Do not enable in production

Development

The log transport is fundamentally very simple, but requires a basic understanding of Node streams- I'd recommend reading this article from Node Source on Understanding Streams in Node.js. All the transport does is spawn a child process to run the passed server, and hook the stream of that child process's standard output into a transform stream, to format the logs for Datadog, and then pipe that transform stream to a write stream which sends what it receives over a UDP connection to the Datadog Agent.

Running the tests

The unit tests can be run with Jest by running npm run test.

The integration tests for the transport check that it is successfully transmitting messages from an emitter process (which just logs a sequence of messages into stdout, standing in for the server) to a listener process (which is a simple UDP socket, standing in for the Datadog Agent). They can be run with npm run test:integration. The test can be controlled with the constants in test/integration/constants.js.