stream_spy

this is just a mad idea to crack one of the node.js pitfalls raised in https://github.com/joyent/node/issues/7804

the aim is to collect more intel on errors thrown in streams. this to make the lives of us poor developers a bit more sweet. we cannot rely any longer on poor stack traces when errors are thrown from streams.

examples

simple

var textStream  = fs.createReadStream('./test/dark.txt'),
    destination = new Stream.Writable(),

    StreamSpy   = require('stream_spy'),
    streamSpy   = new StreamSpy(textStream)

textStream.pipe(destination)

textStream.on('error', function(err) {
    console.error(err.toString())
})

textStream.emit('error', new Error('writable error')) // just emit a fake error

which will output some intel on the source and destination like this:

Error: writable error
<- Source: { constructorName: 'ReadStream',
  path: './test/dark.txt',
  sent: 'darkness\n',
  emittedEvents:
   [ 'open',
     'readable',
     'data',
     'end',
     'error' ] }
-> Destination: { constructorName: 'Writable',
  write: 'function (data) {\n            content += data.toString()\n        }',
  received: 'darkness\n' }

it is really surprising that nodejs core isn't already outputting error context like this. more error context like that definitely helps!

EPIPE example

this example demonstrates the infamous EPIPE scenario:

var streamSpy = new StreamSpy()

http.createServer(function(req, res) {

    req.on('data', function() {
        res.write('pong 1')
        res.end()
    })

    this.close()
}).listen(function() {

    // req is a writable stream
    var req = http.request({
        agent:  false,
        host:   this.address().address,
        port:   this.address().port,
        method: 'POST'
    })

    streamSpy.infiltrate(req)

    req.on('response', function(res) {
        res.on('data', function() {
            req.write('ping 2')
            req.end()
        })
    })

    req.on('error', function(err) {
        console.error(err.toString())
    })

    req.write('ping 1')
})

the infiltrated stream adds additional, very useful, error context to the toString function:

Error: write EPIPE
<- Source: { constructorName: 'ClientRequest',
  path: '/',
  write: 'function (chunk, encoding) {\n  if (!this._header) {\n    this._implic ... unk, encoding);\n  }\n\n  debug(\'write ret = \' + ret);\n  return ret;\n}',
  sent: 'ping 1',
  host: '127.0.0.1',
  port: '44613',
  emittedEvents:
   [ 'socket',
     'drain',
     'response',
     'finish',
     'error' ] }
-> Destination: { constructorName: 'Socket',
  host: '127.0.0.1',
  port: '57112',
  write: 'function (chunk, encoding, cb) {\n  if (typeof chunk !== \'string\' &&  ... );\n  return stream.Duplex.prototype.write.apply(this, arguments);\n}',
  nextDestination: 'IncomingMessage',
  sending: '6\r\nping 2\r\n',
  sent: 'pong 1' }

tips

infiltrate before piping

make sure you infiltrate the stream before it gets piped otherwise you'll miss out valuable intel.

dirty approach to assist you with stream error trouble

if you are stuck with weird stream errors, just wrap all streams with spies in your code with a regex. the most minified code you could use, is:

new require('stream_spy')(stream)

and that's all. any thrown errors within infiltrated streams will contain more intel.

todo

• add more tests (with any kind of streams)
• run heavy stability tests
• enhance robustness
• ignite discussions

useful links

• https://github.com/joyent/node/blob/master/deps/v8/src/messages.js#L1172
• http://www.ecma-international.org/ecma-262/5.1/#sec-15.11