Test-runner

It runs tests. And stuff.

Usage

npm install --save-dev @myndzi/test-runner

package.json:

"scripts": {
    "test": "test-runner"
}

Run tests:

npm test [filters] [-- --switches]

Command line arguments

Most of these will be explained further below.

• -q --quiet - sets log level to 'none'
• -v --verbose - sets log level to 'debug'
• -V --trace - sets log level to 'trace'
• -r --do-report - enables report output
• -c --do-cov - enables coverage report
• -l --do-lint - enables jshint
• -P --perf-only - do a performance test only, no unit tests, etc.
• -L --lint-only - run jshint only, no unit tests, etc.
• --spec - use spec reporter
• --color - enable color
• --bail - stop running tests after first error
• --base-dir - specify the app base dir
• --report-dir - specify the report dir

Any of the boolean switches can be negated by prefixing with no-, for example, --no-color will disable color output. Single-letter arguments can be crammed together in typical Linux fashion, e.g. -rpc

Note:

Since npm can accept arguments itself, it is necessary to distinguish between 'arguments to be given to npm' and 'arguments to be given to test-runner' when using npm to run tests. The usual way to do that sort of thing in Linux is to separate the arguments with --

Therefore, if you wanted to use the npm test approach (and you do, because it's the only helpful way to run a local module without specifying a full path to its binary script!), you must do it like this when specifying switches:

npm test -- -rpc --report-dir=./foo

Thankfully this is not necessy for using filters:

npm test services

The above works just fine. npm >= 2.0 is required for this behavior

Note:

Log level defaults to 'warn'; log level is passed on to all subtasks

Configuration

There are some defaults but they're not that useful. You'll likely want to create ./test/.runner.json; this file configures a few things about where the tests are and where the code is. The current configuration keys are:

• preload - a .js file to load and execute before running any tests
• unload - a .js file to load and execute after running all tests
• perfTest - a .js file to load and execute to run a performance test
• soureDirs - an array of strings relative to <appRoot> for directories that contain source code to instrument for coverage and to lint
• testDirs - an ordered array of strings relative to <appRoot>/test for directories that contain tests to run
• subModules - an array of module names to also run tests on, if possible

Note:

preload and unload scripts should export a function like so:

module.exports = function () {
};

The perfTest script should export a function, and is passed a couple arguments like so:

module.exports = function (baseDir, opts) {
};

Where baseDir is the application base dir (not the test dir), and opts may contain logLevel (a log level threshold), doReport (a boolean, whether to write a report), and reportDir (a string, where to write the report to).

perfTest should return a promise that resolves when the test is complete and the report has been written.

Test ordering

By default, tests exit early at the first failure. Tests are run in the order of the testDirs specified in the config file. Ideally, the fastest tests (e.g. unit tests) come first and the slowest tests (e.g. integration tests) come last; this enables the test run to fail as fast as possible in the case of failure.

If subModules is specified, these tests are run prior to the above. This enables simultaneous development of packages that depend on other packages. test-runner will look for .runner.json in these other packages' own test directories to determine what to do. If submodule tests fail, the process will exit before running any tests of the module proper.

Filtering

To further aid in easily targeting and running specific tests, test-runner implements a filtering system that enables you to run selected subsets of your tests; this is useful for example when you are trying to make a specific test pass and running the entire suite takes a while. To use filtering, simply supply keywords to filter by as arguments:

npm test foo
npm test foo bar baz

Specified filters are intersected -- only tests matching all specified filters will be run.

Filtering is a two-step process involving test location and test filename. All the given filters are compared against submodule names and test directory names first; matches will confine the tests being run to those locations. Any remaining filters are used to filter out specific files by matching against .-separated tokens in the filename.

To give a concrete example, npm test unit would load all test files in the ./test/unit directory, provided it exists and is specified as a testDir in the config. Similarly, npm test logger would run against the logger submodule, provided it was specified in the subModules key of the config, existed (was installed into node_modules), and contained a test directory (possibly with its own .runner.json config file).

Filename-wise, npm test services locations would load all tests in the file ./test/services/locations.test.js but ignore ./test/unit/locations.test.js and ./test/services/categories.test.js

It's a bit complicated to explain, but simple to use; in practice, just include enough keywords to target what you want. You can cross-cut your test directories and target a specific group of functionality to run unit and integration tests, or you can target just unit tests for everything, or just integration tests, and so on.

Reporting

By default, no reports are created. If reporting is enabled, an xUnit compatible xml file is created for the results of the tests and a Cobertura compatible xml file is created for the results of the coverage tests.

Performance test results are up to the specified performance test module to create; sites-backend creates xml files that are usable by the Jenkins Plot plugin.

Programmatic use

var Runner = require('test-runner');
var runner = new Runner(opts, baseDir);
runner.run().then(...).catch(...);

opts are mostly as the command line switches, but are camelCased (--do-cov -> doCov). You can pass subtask-specific options here, too:

var runner = new Runner({
    mocha: {
        logLevel: 'trace',
        doReport: false
    },
    perf: {
        logLevel: 'none',
        doReport: true
    }
});

Valid subtasks are 'mocha', 'coverage', 'perf', and 'lint'