nine-track

Record and playback HTTP requests

This is built to make testing against third party services a breeze. No longer will your test suite fail because an external service is down.

nine-track is inspired by cassette and vcr. This is a fork of eight-track due to permissioning issues.

Breaking changes in 3.0.0

We found a regression when saving series requests in parallel. To remedy this, we are only using pastRequestKeys from the time of the initial request. This means some of your hashes might change for series based requests.

Getting Started

Install the module with: npm install nine-track

// Start up a basic application
var express = require('express');
var nineTrack = require('nine-track');
var request = require('request');
express().use(function (req, res) {
  console.log('Pinged!');
  res.send('Hello World!');
}).listen(1337);

// Create a server using a `nine-track` middleware to the original
express().use(nineTrack({
  url: 'http://localhost:1337',
  fixtureDir: 'directory/to/save/responses'
})).listen(1338);

// Hits original server, triggering a `console.log('Pinged!')` and 'Hello World!' response
request('http://localhost:1338/', console.log);

// Hits saved response but still receieves 'Hello World!' response
request('http://localhost:1338/', console.log);

Documentation

nine-track exposes nineTrack as its module.exports.

nineTrack(options)

Middleware creator for new nineTrack's. This is not a constructor.

• options Object - Container for parameters
  • url String|Object - URL of a server to proxy to
    • If it is a string, it should be the base URL of a server
    • If it is an object, it should be parameters for url.format
  • fixtureDir String - Path to load/save HTTP responses
    • Files will be saved with the format {{method}}_{{encodedUrl}}_{{hashOfRequestContent}}.json
    • An example filename is GET_%2F_658e61f2a6b2f1ae4c127e53f28dfecd.json
  • preventRecording Boolean - Flag to throw errors if a request has not been recorded previously
    • By default, this is false; no errors will be thrown
    • This can be useful in CI to reveal missing fixtures
  • normalizeFn Function - Function to adjust request's save location signature
    • If you would like to make two requests resolve from the same response file, this is how.
    • The function signature should be function (info) and can either mutate the info or return a fresh object
    • info Object - Container for request information
      • httpVersion String - HTTP version received from request (e.g. 1.0, 1.1)
      • headers Object - Headers received by request
        • An example would be {"host": "locahost:1337"}
      • trailers Object - Trailers received by request
      • method String - HTTP method that was used (e.g. GET, POST)
      • url String - Pathname that request arrived from
        • An example would be /
      • body Buffer - Buffered body that was written to request
    • Existing normalizeFn libraries (e.g. multipart/form-data can be found below)
  • scrubFn Function - Functon to adjust request's and response's before saving to disk
    • If you would like to sanitize information from JSON files before saving, this is how.
    • The function signature should be function (info) and can either mutate info or return a fresh object
    • scrubFn is run twice: on hash key lookup and when saving to disk. For key lookup, only info.request is defined and info.response must be checked for before manipulating it
    • info Object - Container for request and response information
      • request Object - Container for request information
        • Same information as present in normalizeFn.info
      • response Object - Container for response information
        • httpVersion String - HTTP version received from response (e.g. 1.0, 1.1)
        • headers Object - Headers received by response
          • An example would be {"x-powered-by": "Express"}
        • trailers Object - Trailers received by response
        • statusCode Number - Status code received from response
          • An example would be 200
        • body Buffer - Body received from response
          • If this is adjusted, we will automatically correct the Content-Length response header

nineTrack returns a middleware with the signature function (req, res)

// Example of string url
nineTrack({
  url: 'http://localhost:1337',
  fixtureDir: 'directory/to/save/responses'
});

// Example of object url
nineTrack({
  url: {
    protocol: 'http:',
    hostname: 'localhost',
    port: 1337
  },
  fixtureDir: 'directory/to/save/responses'
});

If you need to buffer the data before passing it off to nine-track that is supported as well. The requirement is that you record the data as a Buffer or String to req.body.

normalizeFn libraries

• multipart/form-data - Ignore randomly generated boundaries and consolidate similar multipart/form-data requests
  • Website: https://github.com/twolfson/nine-track-normalize-multipart

nineTrack.forwardRequest(req, cb)

Forward an incoming HTTP request in a mikeal/request-like format.

• req http.IncomingMessage - Inbound request to an HTTP server (e.g. from http.createServer)
  • Documentation: http://nodejs.org/api/http.html#http_http_incomingmessage
• cb Function - Callback function with (err, res, body) signature
  • err Error - HTTP error if any occurred (e.g. ECONNREFUSED)
  • res Object - Container that looks like an HTTP object but simiplified due to saving to disk
    • httpVersion String - HTTP version received from external server response (e.g. 1.0, 1.1)
    • headers Object - Headers received by response
    • trailers Object - Trailers received by response
    • statusCode Number - Status code received from external server response
    • body Buffer - Buffered body that was written to response
  • body Buffer - Sugar variable for res.body

nineTrack.startSeries(key)

Begin a series of requests that rely on each other. We will compound past keys onto the hash generated for the current request. This allows for testing items like:

1. Retrieve item, verify non-existence
2. Create item
3. Retrieve item, verify existence

Normally, we would be unable to test this since steps (1) and (3) have the same signature. However, by compounding the previous request keys into our key, we can handle this.

You must remember to run stopSeries() at the end of a series. If you do not, it will pollute future requests and make your tests brittle.

• key String - Namespace to use in hashing our requests
  • This is practical to prevent collisions of similar tests that rely on the same request (e.g. retrieving all resources)

For your convenience, if a series is corrupted (e.g. a request signature changes), then we will attempt clean up the series and require a re-run of your test suite. We do not try to re-run with saved information since states could be inconsistent.

var nineTrackInstance = nineTrack({
  url: {
    protocol: 'http:',
    hostname: 'localhost',
    port: 1337
  },
  fixtureDir: 'directory/to/save/responses'
});
nineTrackInstance.startSeries('create-test');
// Run get, create, get as requests in series
nineTrackInstance.stopSeries();

nineTrack.stopSeries()

Stop a series of requests. This will remove the chaining effect from startSeries and reset nineTrack to default behavior.

Examples

Proxy server with subpath

nine-track can talk to servers that are behind a specific path

// Start up a server that echoes our path
express().use(function (req, res) {
  res.send(req.path);
}).listen(1337);

// Create a server using a `nine-track` middleware to the original
express().use(nineTrack({
  url: 'http://localhost:1337/hello',
  fixtureDir: 'directory/to/save/responses'
})).listen(1338);

// Logs `/hello/world`, concatenated result of `/hello` and `/world` paths
request('http://localhost:1338/world', console.log);

Normalizing requests

Sometimes requests have unpredictable an header or body (e.g. timestamp). We can leverage normalizeFn to make our request hashes consistent to force the same look up.

This does not affect outgoing request data.

// Start up a server that echoes our path
express().use(function (req, res) {
  res.send(req.path);
}).listen(1337);

// Create a server using a `nine-track` middleware to the original
express().use(nineTrack({
  url: 'http://localhost:1337',
  fixtureDir: 'directory/to/save/responses',
  normalizeFn: function (info) {
    if (info.headers['X-Timestamp']) {
      // Normalize all timestamps to a consistent number
      info.headers['X-Timestamp'] = '2015-02-12T00:00:00.000Z';
    }
  }
})).listen(1338);

// On first run, makes valid request
// On future runs, repeats same response
request({
  url: 'http://localhost:1338/world',
  headers: {
    'X-Timestamp': (new Date()).toISOString()
  }
}, console.log);

Scrubbing requests

In some repositories, there is sensitive data being sent/received in requests/responses that you would like to be sanitized. scrubFn takes request/response information and removes it from the saved content and hash.

// Start up a server that echoes our path
express().use(bodyParser.urlencoded()).use(function (req, res) {
  res.send(req.body.sensitive_token === 'password');
}).listen(1337);

// Create a server using a `nine-track` middleware to the original
express().use(nineTrack({
  url: 'http://localhost:1337',
  fixtureDir: 'directory/to/save/responses',
  scrubFn: function (info) {
    var bodyObj = querystring.parse(info.request.body.toString('utf8'));
    if (bodyObj.sensitive_token) {
      // Normalize all sensitive token to a hidden value
      bodyObj.sensitive_token = '****';
      info.request.body = querystring.stringify(bodyObj);
    }
  }
})).listen(1338);

// On first run, makes successful request and saves sanitized data
// On future runs, repeats same response
request({
  url: 'http://localhost:1338/world',
  form: {
    sensitive_token: 'password'
  }
}, console.log); // true

// Saved to disk
/*
"request": {
  "body": "sensitive_token=****"
}
*/

Modifying response data

Occasionally, we want to reply with near accurate data but adjust it slightly (e.g. return an empty list, reproduce an encoding issue). For this example, we will leverage forwardRequest to return an adjusted list.

// Start up a server that echoes our path
express().use(function (req, res) {
  res.json({items: ['a', 'b', 'c']});
}).listen(1337);

// Create a server using a `nine-track` middleware to the original
var nineTrackFn = nineTrack({
  url: 'http://localhost:1337',
  fixtureDir: 'directory/to/save/responses'
});
express().use(function (localReq, localRes) {
  nineTrackFn.forwardRequest(localReq, function (err, remoteRes, remoteBody) {
    // If there was an error, emit it
    if (err) {
      return localReq.emit('error', err);
    }

    // Otherwise, attempt to adjust the body
    var remoteJson = JSON.parse(remoteBody);
    if (remoteJson.items.length === 3) {
      remoteJson.items.pop();
    }

    // Send our response
    localRes.json(remoteJson);
  });
}).listen(1338);

// On first run, makes successful request and saves sanitized data
// On future runs, repeats same response
request('http://localhost:1338/world', console.log); // {items: ['a', 'b']}

// Saved on disk
/*
"response": {
  "body": "{\n  \"items\": [\n    \"a\",\n    \"b\",\n    \"c\"\n  ]\n}"
}
*/

Prevent recording

In CI, it can be useful to prevent requests to remote servers since all fixtures should be saved by this point. In this example, we leverage preventRecording and the Travis CI environment variable to not allow new requests in Travis CI.

// Start up a server that echoes our path
express().use(function (req, res) {
  res.json(req.path);
}).listen(1337);

// Create a server using a `nine-track` middleware to the original
express().use(nineTrack({
  url: 'http://localhost:1337',
  fixtureDir: 'directory/to/save/responses',
  preventRecording: !!process.env.TRAVIS
})).listen(1338);

request('http://localhost:1338/world', console.log);

// On an unsaved fixture in "Travis CI"
/*
events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: Fixture not found for request "{"httpVersion":"1.1","headers":{"host":"localhost:1338","connection":"keep-alive"},"trailers":{},"method":"GET","url":"/world","body":""}"
    at createRemoteReq (/home/todd/github/nine-track/lib/nine-track.js:240:21)
    at fn (/home/todd/github/nine-track/node_modules/async/lib/async.js:582:34)
    at Object._onImmediate (/home/todd/github/nine-track/node_modules/async/lib/async.js:498:34)
    at processImmediate [as _immediateCallback] (timers.js:354:15)
*/

Contributing

In lieu of a formal styleguide, take care to maintain the existing coding style. Add unit tests for any new or changed functionality. Lint via npm run lint and test via npm test.

License

All work up to and including 87a024b is owned by Uber under the MIT license.

After that commit, all modifications to the work have been released under the UNLICENSE to the public domain.