npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

bunyan-gcp-logging

v0.1.0

Published

Bunyan Transport for GCP (Google Cloud Platform) using Cloud Logging

Downloads

4,780

Readme

Bunyan GCP Cloud Logging Transport

Simple Bunyan transport to leverage Google Cloud Platform configuration for Cloud Logging

Usage

Just configure this transport for the logger:

const Bunyan = require('bunyan');
const {createStream} = require('bunyan-gcp-logging');

const logger = Bunyan.createLogger({
  name: 'MyLogName',
  streams: [createStream()],
});

Optionally, you can configure the log level for the transport and the final output stream (stdout by default).

const Bunyan = require('bunyan');
const {createStream} = require('bunyan-gcp-logging');

const logger = Bunyan.createLogger({
  name: 'warnings',
  streams: [createStream(Bunyan.WARN, process.stderr)],
});

The Problem

If your node application is deployed on GCP (Google Cloud Platform), and you've enabled logging for it (the default); all your logs will end up in Cloud Logging Viewer but won't be very readable, and won't be using most of Cloud Logging features.

Bunyan uses structured logging, but with a different schema than Cloud Logging expect to make take advantange of it's features.

Some of the features enabled by using this module:

  • Show the correct severity (level) in Cloud Logging
  • Show the msg in the summary. (Cloud Logging expects a message key instead of msg)
  • Render information about the request.
  • Track errors in Cloud Logging Error Reporting

How does this work?

When using bunyan you can configure transports (streams in reality). If no one is configured, logs go to stdout. In our case, we want them to go to stdout but with a different format. A format that is compatible with what Cloud Logging has understading.

So, in essence all you need to do is configure this module as the stream.

const Bunyan = require('bunyan');
const {createStream} = require('bunyan-gcp-logging');

const logger = Bunyan.createLogger({
  name: 'MyLogName',
  streams: [createStream()],
});

The lifecycle of a log entry in GKE

So, what happens when you do logger.info('hello world')?

First, bunyan creates a log record something like {v: 1, level: 30, msg: 'hello world', ...}. Then, is passed to the configured stream, which if none was configured, is simply sending all to stdout

Second, since you are running on a docker container within a node in GCP; docker will wrap that log entry into something like

{
  "stream": "stdout",
  "time": "2018-08-24T12:41:50.987184687Z",
  "log": "{\"level\":30,\"time\":1535114510986,\"msg\":\"hello world\"}"
}

Lastly, since you have logging configured for GCP, there a fluentd daemon on each node. fluentd aggregates logs from all your containers, transform them based on it's configuration, and finally exports them. On GCP, fluentd is configured to export log entries to Cloud Logging. Also, it's configured to unwrap the docker log entry and parse your original entry. It will recognize some fields as part of the Cloud Logging schema and use them. But mostly it will pass all of them to Cloud Logging.

The expected log entry

So, to be a good citizen with Cloud Logging and the fluentd configuration in GCP, we need to make a few changes to the original entry.

  1. Use message instead of msg
  2. When logging an error, use err.stack as the value for message
  3. When logging an http request, use httpRequest to log the details about it
  4. Map level to severity

For more information about the stackdriver log entry schema check:

  • General: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
  • For errrors: https://cloud.google.com/error-reporting/docs/formatting-error-messages

Log Trasformation

When in bunyan you log like:

logger.info({data: {from: 'me', to: 'you'}}, 'data transfer');

Bunyan will generate

{
  "name": "src-example",
  "level": 30,
  "msg": "data transfer",
  "data": {"from": "me", "to": "you"},
  "time": "2012-02-06T04:19:35.605Z",
  "src": {
    "file": "/Users/trentm/tm/node-bunyan/examples/src.js",
    "line": 20,
    "func": "Wuzzle.woos"
  },
  "hostname": "banana.local",
  "pid": 123,
  "v": 0
}

To make Cloud Logging compatible it will be transformed into:

{
  "name": "src-example",
  "severity": "INFO",
  "message": "data transfer",
  "data": {"from": "me", "to": "you"},
  "time": "2012-02-06T04:19:35.605Z",
  "src": {
    "file": "/Users/trentm/tm/node-bunyan/examples/src.js",
    "line": 20,
    "func": "Wuzzle.woos"
  }
}

For Errors

when logging an error with bunyan:

log.info(err); // Special case to log an `Error` instance to the record.
// This adds an "err" field with exception details
// (including the stack) and sets "msg" to the exception
// message.
log.info(err, 'more on this: %s', more);
// ... or you can specify the "msg".

Bunyan by default, will keep your error in err key. We will use the err.stack and set it as the message, since that what's required by Cloud Logging.

Also, the error log entry for Cloud Logging requires us to set:

  "serviceContext": {
    "service": string,     // Required.
    "version": string
  },

We will use the name configured as serviceContext.name.

Thanks to this, you can track your error ocurrences in https://cloud.google.com/error-reporting/

Other options

You can bypass fluentd & docker all together and log directly by calling the Cloud Logging API. I'm not pro using a http transport for log within the applicaton, since every application will be doing it's own buffering and sending, plus you will be loosing context information like the node & pod id.

But, if that's not a proble, simply use https://github.com/googleapis/nodejs-logging-bunyan.

Thanks!

Special thanks to Mariano Cortesi for initial efforts. This repository it's based on the bunyan-gke-stackdriver.