npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

logfmt

v1.4.0

Published

key=value logger and parser

Downloads

113,759

Readme

node-logfmt

Build Status

"logfmt" is the name for a key value logging convention we've adopted at Heroku.

This library is for both converting lines in logfmt format to objects and for logging objects to a stream in logfmt format.

It provides a logfmt parser, logfmt stringifier, a logging facility, and both streaming and non-streaming body parsers for express and restify.

You should use this library if you're trying to write structured logs or if you're consuming them (especially if you're writing a logplex drain).

install

npm install logfmt

use

node.js

The logfmt module is a singleton that works directly from require.

var logfmt = require('logfmt');

logfmt.stringify({foo: 'bar'});
// 'foo=bar'

logfmt.parse('foo=bar');
// {foo: 'bar'}

It is also a constructor function, so you can use new logfmt to create a new logfmt that you can configure differently.

var logfmt2 = new logfmt;

// replace our stringify with JSON's
logfmt2.stringify = JSON.stringify

// now we log JSON!
logfmt2.log({foo: 'bar'})
// {"foo":"bar"}

// and the original logfmt is untouched
logfmt.log({foo: 'bar'})
// foo=bar

command line

logfmt

accepts lines on STDIN and converts them to json

> echo "foo=bar a=14 baz=\"hello kitty\" cool%story=bro f %^asdf" | logfmt
{ "foo": "bar", "a": 14, "baz": "hello kitty", "cool%story": "bro", "f": true, "%^asdf": true }

logfmt -r (reverse)

accepts JSON on STDIN and converts them to logfmt

> echo '{ "foo": "bar", "a": 14, "baz": "hello kitty", \
"cool%story": "bro", "f": true, "%^asdf": true }' | logfmt -r
foo=bar a=14 baz="hello kitty" cool%story=bro f=true %^asdf=true

round trips for free!

> echo "foo=bar a=14 baz=\"hello kitty\" cool%story=bro f %^asdf" | logfmt | logfmt -r | logfmt
{ "foo": "bar", "a": 14, "baz": "hello kitty", "cool%story": "bro", "f": true, "%^asdf": true }

API

stringifying

Serialize an object to logfmt format

logfmt.stringify(object)

Serializes a single object.

logfmt.stringify({foo: "bar", a: 14, baz: 'hello kitty'})
//> 'foo=bar a=14 baz="hello kitty"'

parsing

Parse a line in logfmt format

logfmt.parse(string)

logfmt.parse("foo=bar a=14 baz=\"hello kitty\" cool%story=bro f %^asdf code=H12")
//> { "foo": "bar", "a": '14', "baz": "hello kitty", "cool%story": "bro", "f": true, "%^asdf": true, "code" : "H12" }

The only conversions are from the strings true and false to their proper boolean counterparts.

We cannot arbitrarily convert numbers because that will drop precision for numbers that require more than 32 bits to represent them.

Streaming

Put this in your pipe and smoke it.

logfmt.streamParser()

Creates a streaming parser that will automatically split and parse incoming lines and emit javascript objects.

Stream in from STDIN

process.stdin.pipe(logfmt.streamParser())

Or pipe from an HTTP request

req.pipe(logfmt.streamParser())

logfmt.streamStringify([options])

Pipe objects into the stream and it will write logfmt. You can customize the delimiter via the options object, which defaults to \n (newlines).

  var parseJSON = function(line) {
    if(!line) return;
    this.queue(JSON.parse(line.trim()))
  }

  process.stdin
    .pipe(split())
    .pipe(through(parseJSON))
    .pipe(logfmt.streamStringify())
    .pipe(process.stdout)

Example

Example command line of parsing logfmt and echoing objects to STDOUT:

var logfmt  = require('logfmt');
var through = require('through');

process.stdin
  .pipe(logfmt.streamParser())
  .pipe(through(function(object){
    console.log(object);
  }))

Example HTTP request parsing logfmt and echoing objects to STDOUT:

var http    = require('http');
var logfmt  = require('logfmt');
var through = require('through');

http.createServer(function (req, res) {
  req.pipe(logfmt.streamParser())
     .pipe(through(function(object){
       console.log(object);
     }))

  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('OK');
}).listen(3000);

express/restify parsing middleware

  // streaming
  app.use(logfmt.bodyParserStream());
  // buffering
  app.use(logfmt.bodyParser());

logfmt.bodyParserStream([opts])

Valid Options:

  • contentType: defaults to application/logplex-1

If you use the logfmt.bodyParserStream() for a body parser, you will have a req.body that is a readable stream.

Pipes FTW:

var app    = require('express')();
var http   = require('http');
var through = require('through');
var logfmt  = require('logfmt');

app.use(logfmt.bodyParserStream());

app.post('/logs', function(req, res){
  if(!req.body) return res.send('OK');

  req.body.pipe(through(function(line){
    console.dir(line);
  }))

  res.send('OK');
})

http.createServer(app).listen(3000);

Or you can just use the readable event:

var app    = require('express')();
var http   = require('http');
var logfmt  = require('logfmt');

app.use(logfmt.bodyParserStream());

// req.body is now a Readable Stream
app.post('/logs', function(req, res){
  req.body.on('readable', function(){
    var parsedLine = req.body.read();
    if(parsedLine) console.log(parsedLine);
    else res.send('OK');
  })
})

http.createServer(app).listen(3000);

Non-Streaming

logfmt.bodyParser([opts])

Valid Options:

  • contentType: defaults to application/logplex-1

If you use the logfmt.bodyParser() for a body parser, you will have a req.body that is an array of objects.

var logfmt   = require('logfmt');

app.use(logfmt.bodyParser());

// req.body is now an array of objects
app.post('/logs', function(req, res){
  console.log('BODY: ' + JSON.stringify(req.body));
  req.body.forEach(function(data){
    console.log(data);
  });
  res.send('OK');
})

http.createServer(app).listen(3000);

test it:

curl -X POST --header 'Content-Type: application/logplex-1' -d "foo=bar a=14 baz=\"hello kitty\" cool%story=bro f %^asdf" http://localhost:3000/logs

logging

Log an object to logfmt.stream (defaults to STDOUT)

Uses the logfmt.stringify function to write the result to logfmt.stream

logfmt.log({foo: "bar", a: 14, baz: 'hello kitty'})
//=> foo=bar a=14 baz="hello kitty"
//> undefined

logfmt.log(object, [stream])

Defaults to logging to process.stdout

logfmt.log({ "foo": "bar", "a": 14, baz: 'hello kitty'})
//=> foo=bar a=14 baz="hello kitty"

customizing logging location

logfmt.log() Accepts as 2nd argument anything that responds to write(string)

var logfmt = require('logfmt');
logfmt.log({ "foo": "bar", "a": 14, baz: 'hello kitty'}, process.stderr)
//=> foo=bar a=14 baz="hello kitty"

Overwrite the default global location by setting logfmt.stream

var logfmt = require('logfmt');
logfmt.stream = process.stderr

logfmt.log({ "foo": "bar", "a": 14, baz: 'hello kitty'})
//=> foo=bar a=14 baz="hello kitty"

You can have multiple, isolated logfmts by using new.

var logfmt = require('logfmt');
var errorLogger = new logfmt;
errorLogger.stream = process.stderr

logfmt.log({hello: 'stdout'});
//=> hello=stdout

errorLogger.log({hello: 'stderr'});
//=> hello=stderr

logfmt.namespace(object)

Returns a new logfmt with object's data included in every log call.

var logfmt = require('logfmt').namespace({app: 'logfmt'});

logfmt.log({ "foo": "bar", "a": 14, baz: 'hello kitty'})
//=> app=logfmt foo=bar a=14 baz="hello kitty"

logfmt.log({})
//=> app=logfmt

logfmt.log({hello: 'world'})
//=> app=logfmt hello=world

logfmt.time([label])

Log how long something takes. Returns a new logfmt with elapsed milliseconds included in every log call.

  • label: optional name for the milliseconds key. defaults to: elapsed=<milliseconds>ms
var timer = logfmt.time();
timer.log();
//=> elapsed=1ms

String label changes the key to <string>=<milliseconds>ms

var timer = logfmt.time('time');
timer.log();
//=> time=1ms
timer.log();
//=> time=2ms

If you'd like to include data, just chain a call to namespace.

var timer = logfmt.time('time').namespace({foo: 'bar'});
timer.log();
//=> time=1ms foo=bar
timer.log();
//=> time=2ms foo=bar

logfmt.error(error)

Accepts a Javascript Error object and converts it to logfmt format.

It will print up to logfmt.maxErrorLines lines.

var logfmt = require('logfmt');
logfmt.error(new Error('test error'));
//=> at=error id=12345 message="test error"
//=> at=error id=12345 line=0 trace="Error: test error"
//=> ...

express/restify logging middleware

app.use(logfmt.requestLogger());
//=> ip=127.0.0.1 time=2013-08-05T20:50:19.216Z method=POST path=/logs status=200 content_length=337 content_type=application/logplex-1 elapsed=4ms

logfmt.requestLogger([options], [formatter(req, res)])

If no formatter is supplied it will default to logfmt.requestLogger.commonFormatter which is based on having similar fields to the Apache Common Log format.

Valid Options:

  • immediate: log before call to next() (ie: before the request finishes)
  • elapsed: renames the elapsed key to a key of your choice when in non-immediate mode

Defaults to immediate: true and elapsed: 'elapsed'

app.use(logfmt.requestLogger({immediate: true}, function(req, res){
  return {
    method: req.method
  }
}));
//=> method=POST
app.use(logfmt.requestLogger({elapsed: 'request.time'}, function(req, res){
  return {
    "request.method": req.method
  }
}));
//=> request.method=POST request.time=12ms
formatter(req, res)

A formatter takes the request and response and returns a JSON object for logfmt.log

app.use(logfmt.requestLogger(function(req, res){
  return {
    method: req.method
  }
}));
//=> method=POST elapsed=4ms

It's always possible to piggyback on top of the commonFormatter

app.use(logfmt.requestLogger(function(req, res){
  var data = logfmt.requestLogger.commonFormatter(req, res)
  return {
    ip: data.ip,
    time: data.time,
    foo: 'bar'
  };
}));
//=> ip=127.0.0.1 time=2013-08-05T20:50:19.216Z foo=bar elapsed=4ms

Development

Pull Requests welcome.

Tests

> npm test

License

MIT