metatests

metatests is an extremely simple to use test framework and runner for Metarhia technology stack built on the following principles:

• Test cases are files, tests are either imperative (functions) or declarative (arrays and structures).

• Assertions are done using the built-in Node.js assert module. The framework also provides additional testing facilities (like spies).

• Tests can be run in parallel.

• All tests are executed in isolated sandboxes. The framework allows to easily mock modules required by tests and provides ready-to-use mocks for timers and other core functionality.

• Testing asynchronous operations must be supported.

• Testing pure functions without asynchronous operations and state can be done without extra boilerplate code using DSL based on arrays.

  mt.case(
    'Test common.duration',
    { common },
    {
      // ...
      'common.duration': [
        ['1d', 86400000],
        ['10h', 36000000],
        ['7m', 420000],
        ['13s', 13000],
        ['2d 43s', 172843000],
        // ...
      ],
      // ...
    },
  );

  (Prior art)

• The framework must work in Node.js and browsers (using Webpack or any other module bundler that supports CommonJS modules and emulates Node.js globals).

Contributors

• See github for full contributors list

API

• Interface metatests
  • case
  • DeclarativeTest
    • DeclarativeTest.prototype.constructor
    • DeclarativeTest.prototype.run
    • DeclarativeTest.prototype.runNow
  • equal
  • strictEqual
  • reporters.Reporter
    • reporters.Reporter.prototype.constructor
    • reporters.Reporter.prototype.error
    • reporters.Reporter.prototype.finish
    • reporters.Reporter.prototype.log
    • reporters.Reporter.prototype.logComment
    • reporters.Reporter.prototype.record
  • reporters.ConciseReporter
    • reporters.ConciseReporter.prototype.constructor
    • reporters.ConciseReporter.prototype.error
    • reporters.ConciseReporter.prototype.finish
    • reporters.ConciseReporter.prototype.listFailure
    • reporters.ConciseReporter.prototype.parseTestResults
    • reporters.ConciseReporter.prototype.printAssertErrorSeparator
    • reporters.ConciseReporter.prototype.printSubtestSeparator
    • reporters.ConciseReporter.prototype.printTestSeparator
    • reporters.ConciseReporter.prototype.record
  • reporters.TapReporter
    • reporters.TapReporter.prototype.constructor
    • reporters.TapReporter.prototype.error
    • reporters.TapReporter.prototype.finish
    • reporters.TapReporter.prototype.listFailure
    • reporters.TapReporter.prototype.logComment
    • reporters.TapReporter.prototype.parseTestResults
    • reporters.TapReporter.prototype.record
  • runner.Runner
    • runner.Runner.prototype.constructor
    • runner.Runner.prototype.addTest
    • runner.Runner.prototype.finish
    • runner.Runner.prototype.removeReporter
    • runner.Runner.prototype.resume
    • runner.Runner.prototype.runTodo
    • runner.Runner.prototype.setReporter
    • runner.Runner.prototype.wait
  • runner.instance
  • speed
  • measure
  • convertToCsv
  • ImperativeTest
    • ImperativeTest.prototype.constructor
    • ImperativeTest.prototype.afterEach
    • ImperativeTest.prototype.assert
    • ImperativeTest.prototype.assertNot
    • ImperativeTest.prototype.bailout
    • ImperativeTest.prototype.beforeEach
    • ImperativeTest.prototype.case
    • ImperativeTest.prototype.cb
    • ImperativeTest.prototype.cbFail
    • ImperativeTest.prototype.contains
    • ImperativeTest.prototype.containsGreedy
    • ImperativeTest.prototype.defer
    • ImperativeTest.prototype.doesNotThrow
    • ImperativeTest.prototype.end
    • ImperativeTest.prototype.endAfterSubtests
    • ImperativeTest.prototype.equal
    • ImperativeTest.prototype.error
    • ImperativeTest.prototype.fail
    • ImperativeTest.prototype.is
    • ImperativeTest.prototype.isArray
    • ImperativeTest.prototype.isBuffer
    • ImperativeTest.prototype.isError
    • ImperativeTest.prototype.isRejected
    • ImperativeTest.prototype.isResolved
    • ImperativeTest.prototype.mustCall
    • ImperativeTest.prototype.mustNotCall
    • ImperativeTest.prototype.notEqual
    • ImperativeTest.prototype.notOk
    • ImperativeTest.prototype.notSameTopology
    • ImperativeTest.prototype.ok
    • ImperativeTest.prototype.on
    • ImperativeTest.prototype.pass
    • ImperativeTest.prototype.plan
    • ImperativeTest.prototype.regex
    • ImperativeTest.prototype.rejects
    • ImperativeTest.prototype.resolves
    • ImperativeTest.prototype.run
    • ImperativeTest.prototype.same
    • ImperativeTest.prototype.sameTopology
    • ImperativeTest.prototype.strictEqual
    • ImperativeTest.prototype.strictNotSame
    • ImperativeTest.prototype.strictSame
    • ImperativeTest.prototype.test
    • ImperativeTest.prototype.testAsync
    • ImperativeTest.prototype.testSync
    • ImperativeTest.prototype.throws
    • ImperativeTest.prototype.type
  • test
  • testSync
  • testAsync

Interface: metatests

case(caption, namespace, list, runner)

• caption: <string> case caption
• namespace: <Object> namespace to use in this case test
• list: <Object> hash of <Array>, hash keys are function and method names. <Array> contains call parameters last <Array> item is an expected result (to compare) or <Function> (pass result to compare)
• runner: <Runner> runner for this case test, optional, default: metatests.runner.instance

Create declarative test

class DeclarativeTest extends Test

DeclarativeTest.prototype.constructor(caption, namespace, list, options)

DeclarativeTest.prototype.run()

DeclarativeTest.prototype.runNow()

equal(val1, val2)

strictEqual(val1, val2)

class reporters.Reporter

reporters.Reporter.prototype.constructor(options)

• options: <Object>
  • stream: <stream.Writable> optional

reporters.Reporter.prototype.error(test, error)

• test: <Test>
• error: <Error>

Fail test with error

reporters.Reporter.prototype.finish()

reporters.Reporter.prototype.log(...args)

reporters.Reporter.prototype.logComment(...args)

reporters.Reporter.prototype.record(test)

• test: <Test>

Record test

class reporters.ConciseReporter extends Reporter

reporters.ConciseReporter.prototype.constructor(options)

reporters.ConciseReporter.prototype.error(test, error)

reporters.ConciseReporter.prototype.finish()

reporters.ConciseReporter.prototype.listFailure(test, res, message)

reporters.ConciseReporter.prototype.parseTestResults(test, subtest)

reporters.ConciseReporter.prototype.printAssertErrorSeparator()

reporters.ConciseReporter.prototype.printSubtestSeparator()

reporters.ConciseReporter.prototype.printTestSeparator()

reporters.ConciseReporter.prototype.record(test)

class reporters.TapReporter extends Reporter

reporters.TapReporter.prototype.constructor(options)

reporters.TapReporter.prototype.error(test, error)

reporters.TapReporter.prototype.finish()

reporters.TapReporter.prototype.listFailure(test, res, offset)

reporters.TapReporter.prototype.logComment(...args)

reporters.TapReporter.prototype.parseTestResults(test, offset = 0)

reporters.TapReporter.prototype.record(test)

class runner.Runner extends EventEmitter

runner.Runner.prototype.constructor(options)

runner.Runner.prototype.addTest(test)

runner.Runner.prototype.finish()

runner.Runner.prototype.removeReporter()

runner.Runner.prototype.resume()

runner.Runner.prototype.runTodo(active = true)

runner.Runner.prototype.setReporter(reporter)

runner.Runner.prototype.wait()

runner.instance

• <Runner>

speed(caption, count, cases)

• caption: <string> name of the benchmark
• count: <number> amount of times ro run each function
• cases: <Array> functions to check

Microbenchmark each passed function and compare results.

measure(cases[, options])

• cases: <Array> cases to test, each case contains
  • fn: <Function> function to check, will be called with each args provided
  • name: <string> case name, function.name by default
  • argCases: <Array> array of arguments to create runs with. When omitted fn will be run once without arguments. Total amount of runs will be runs * argCases.length.
  • n: <number> number of times to run the test, defaultCount from options by default
• options: <Object>
  • defaultCount: <number> number of times to run the function by default, default: 1e6
  • runs: <number> number of times to run the case, default: 20
  • preflight: <number> number of times to pre-run the case for each set of arguments, default: 10
  • preflightCount: <number> number of times to run the function in the preflight stage, default: 1e4
  • listener: <Object> appropriate function will be called to report events, optional
    • preflight: <Function> called when preflight is starting, optional
      • name: <string> case name
      • count: <number> number of times it will be run
      • args: <Array> function arguments
    • run: <Function> called when run is starting, optional
      • name: <string> case name
      • count: <number> number of times it will be run
      • args: <Array> function arguments
    • cycle: <Function> called when run is done, optional
      • name: <string> case name
      • result: <Object> case results
    • done: <Function> called when all runs for given configurations are done, optional
      • name: <string> case name
      • args: <Array> current configuration
      • results: <Array> results of all runs with this configuration
    • finish: <Function> called when measuring is finished, optional
      • results: <Array> all case results

Returns: <Array> results of all cases as objects of structure

• name: <string> case name
• args: <Array> arguments for this run
• count: <number> number of times case was run
• time: <number> time in nanoseconds it took to make count runs
• result: <any> result of one of the runs

Microbenchmark each passed configuration multiple times

convertToCsv(results)

• results: <Array> all results from measure run

Returns: <string> valid CSV representation of the results

Convert metatests.measure result to csv.

class ImperativeTest extends Test

ImperativeTest.prototype.constructor(caption, func, options)

ImperativeTest.prototype.afterEach(func)

• func: <Function>
  • subtest: <ImperativeTest> test instance
  • callback: <Function>
  • Returns: <Promise>|<void>

Set a function to run after each subtest.

The function must either return a promise or call a callback.

ImperativeTest.prototype.assert(value[, message])

• value: <any> value to check
• message: <string> description of the check, optional

Check if value is truthy.

ImperativeTest.prototype.assertNot(value[, message])

• value: <any> value to check
• message: <string> description of the check, optional

Check if value is falsy.

ImperativeTest.prototype.bailout([err][, message])

• err: <Error> bailout error
• message: <string> bailout message

Fail this test and throw an error.

If both err and message are provided err.toString() will be appended to message.

ImperativeTest.prototype.beforeEach(func)

• func: <Function>
  • subtest: <ImperativeTest> test instance
  • callback: <Function>
    • context: <any> context of the test. It will pe passed as a second argument to test function and is available at test.context
  • Returns: <Promise>|<void> nothing or Promise resolved with context

Set a function to run before each subtest.

The function must either return a promise or call a callback.

ImperativeTest.prototype.case(message, namespace, list, options = {})

Create a declarative case() subtest of this test.

ImperativeTest.prototype.cb([msg][, cb])

• msg: <string> test.error message
• cb: <Function> callback function

Returns: <Function> function to pass to callback

Create error-first callback wrapper to perform automatic checks.

This will check for test.mustCall() the callback and {test.error()} the first callback argument.

ImperativeTest.prototype.cbFail([fail][, cb[, afterAllCb]])

• fail: <string> test.fail message
• cb: <Function> callback function to call if there was no error
• afterAllCb: <Function> function called after callback handling

Returns: <Function> function to pass to callback

Create error-first callback wrapper to fail test if call fails.

This will check for test.mustCall() the callback and if the call errored will use test.fail() and test.end()

ImperativeTest.prototype.contains(actual, subObj[, message[, sort[, test]]])

• actual: <any> actual data
• subObj: <any> expected properties
• message: <string> description of the check, optional
• sort: <boolean | Function> if true or a sort function sort data properties, default: false
• cmp: <Function> test function, default: compare.strictEqual
  • actual: <any>
  • expected: <any>
  • Returns: <boolean> true if actual is equal to expected, false otherwise

Check that actual contains all properties of subObj.

Properties will be compared with test function.

ImperativeTest.prototype.containsGreedy(actual, subObj[, message[, sort[, test]]])

• actual: <any> actual data
• subObj: <any> expected properties
• message: <string> description of the check, optional
• cmp: <Function> test function, default: compare.strictEqual
  • actual: <any>
  • expected: <any>
  • Returns: <boolean> true if actual is equal to expected, false otherwise

Check greedily that actual contains all properties of subObj.

Similar to test.contains() but will succeed if at least one of the properties in actual match the one in subObj.

ImperativeTest.prototype.defer(fn, options)

• fn: <Function> function to call before the end of test. Can return a promise that will defer the end of test.
• options: <Object>
  • ignoreErrors: <boolean> ignore errors from fn function, default: false

Defer a function call until the 'before' end of test.

ImperativeTest.prototype.doesNotThrow(fn[, message])

• fn: <Function> function to run
• message: <string> description of the check, optional

Check that fn doesn't throw.

ImperativeTest.prototype.end()

Finish the test.

This will fail if the test has unfinished subtests or plan is not complete.

ImperativeTest.prototype.endAfterSubtests()

Mark this test to call end after its subtests are done.

ImperativeTest.prototype.equal(actual, expected[, message])

• actual: <any> actual data
• expected: <any> expected data
• message: <string> description of the check, optional

Compare actual and expected for non-strict equality.

ImperativeTest.prototype.error(err[, message])

• err: <any> error to check
• message: <string> description of the check, optional

Fail if err is instance of Error.

ImperativeTest.prototype.fail([message][, err])

• message: <string | Error> failure message or error, optional
• err: <Error> error, optional

Fail this test recording failure message.

This doesn't call test.end().

ImperativeTest.prototype.is(checkFn, val[, message])

• checkFn: <Function> condition function
  • val: <any> provided value
• Returns: <boolean> true if condition is satisfied and false otherwise
• val: <any> value to check the condition against
• message: <string> check message, optional

Check whether val satisfies custom checkFn condition.

ImperativeTest.prototype.isArray(val[, message])

• val: <any> value to check
• message: <string> check message, optional

Check if val satisfies Array.isArray.

ImperativeTest.prototype.isBuffer(val[, message])

• val: <any> value to check
• message: <string> check message, optional

Check if val satisfies Buffer.isBuffer.

ImperativeTest.prototype.isError(actual[, expected[, message]])

• actual: <any> actual error to compare
• expected: <any> expected error, default: new Error()
• message: <string> description of the check, optional

Check if actual is equal to expected error.

ImperativeTest.prototype.isRejected(input, err)

• input: <Promise | Function> promise of function returning thenable
• err: <any> value to be checked with test.isError() against rejected value

Check that input rejects.

ImperativeTest.prototype.isResolved(input[, expected])

• input: <Promise | Function> promise of function returning thenable
• expected: <any> if passed it will be checked with test.strictSame() against resolved value

Verify that input resolves.

ImperativeTest.prototype.mustCall([fn[, count[, name]]])

• fn: <Function> function to be checked, default: () => {}
• count: <number> amount of times fn must be called, default: 1
• name: <string> name of the function, default: 'anonymous'

Returns: <Function> function to check with, will forward all arguments to fn, and result from fn

Check that fn is called specified amount of times.

ImperativeTest.prototype.mustNotCall([fn[, name]])

• fn: <Function> function to not be checked, default: () => {}
• name: <string> name of the function, default: 'anonymous'

Returns: <Function> function to check with, will forward all arguments to fn, and result from fn

Check that fn is not called.

ImperativeTest.prototype.notEqual(actual, expected[, message])

• actual: <any> actual data
• expected: <any> expected data
• message: <string> description of the check, optional

Compare actual and expected for non-strict not-equality.

ImperativeTest.prototype.notOk(value[, message])

• value: <any> value to check
• message: <string> description of the check, optional

Check if value is falsy.

ImperativeTest.prototype.notSameTopology(obj1, obj2[, message])

• obj1: <any> actual data
• obj2: <any> expected data
• message: <string> description of the check, optional

Compare actual and expected to not have the same topology.

ImperativeTest.prototype.ok(value[, message])

• value: <any> value to check
• message: <string> description of the check, optional

Check if value is truthy.

ImperativeTest.prototype.on(name, listener)

ImperativeTest.prototype.pass([message])

• message: <string> message to record

Record a passing assertion.

ImperativeTest.prototype.plan(n)

• n: <number> amount of assertions

Plan this test to have exactly n assertions and end test after

this amount of assertions is reached.

ImperativeTest.prototype.regex(regex, input[, message])

• regex: <RegExp>|<string> pattern to match
• input: <string> input to check
• message: <string>

Test whether input matches the provided RegExp.

ImperativeTest.prototype.rejects(input, err)

• input: <Promise | Function> promise of function returning thenable
• err: <any> value to be checked with test.isError() against rejected value

Check that input rejects.

ImperativeTest.prototype.resolves(input[, expected])

• input: <Promise | Function> promise of function returning thenable
• expected: <any> if passed it will be checked with test.strictSame() against resolved value

Verify that input resolves.

ImperativeTest.prototype.run()

Start running the test.

ImperativeTest.prototype.same(actual, expected[, message])

• actual: <any> actual data
• expected: <any> expected data
• message: <string> description of the check, optional

Compare actual and expected for non-strict equality.

ImperativeTest.prototype.sameTopology(obj1, obj2[, message])

• obj1: <any> actual data
• obj2: <any> expected data
• message: <string> description of the check, optional

Compare actual and expected to have same topology.

Useful for comparing objects with circular references for equality.

ImperativeTest.prototype.strictEqual(actual, expected[, message])

• actual: <any> actual data
• expected: <any> expected data
• message: <string> description of the check, optional

Compare actual and expected for strict equality.

ImperativeTest.prototype.strictNotSame(actual, expected[, message])

• actual: <any> actual data
• expected: <any> expected data
• message: <string> description of the check, optional

Compare actual and expected for strict non-equality.

ImperativeTest.prototype.strictSame(actual, expected[, message])

• actual: <any> actual data
• expected: <any> expected data
• message: <string> description of the check, optional

Compare actual and expected for strict equality.

ImperativeTest.prototype.test(caption, func, options)

• caption: <string> name of the test
• func: <Function> test function
  • test: <ImperativeTest> test instance
• options: <TestOptions>
  • run: <boolean> auto start test, default: true
  • async: <boolean> if true do nothing, if false auto-end test on nextTick after func run, default: true
  • timeout: <number> time in milliseconds after which test is considered timeouted.
  • parallelSubtests: <boolean> if true subtests will be run in parallel, otherwise subtests are run sequentially, default: false
  • dependentSubtests: <boolean> if true each subtest will be executed sequentially in order of addition to the parent test short-circuiting if any subtest fails, default: false

Returns: <ImperativeTest> subtest instance

Create a subtest of this test.

If the subtest fails this test will fail as well.

ImperativeTest.prototype.testAsync(message, func, options = {})

Create an asynchronous subtest of this test.

Simple wrapper for test.test() setting async option to true.

ImperativeTest.prototype.testSync(message, func, options = {})

Create a synchronous subtest of this test

Simple wrapper for test.test() setting async option to false.

ImperativeTest.prototype.throws(fn[, expected[, message]])

• fn: <Function> function to run
• expected: <any> expected error, default: new Error()
• message: <string> description of the check, optional

Check that fn throws expected error.

ImperativeTest.prototype.type(obj, type[, message])

• obj: <any> value to check
• type: <string | Function> class or class name to check
• message: <string> description of the check, optional

Check if obj is of specified type.

test(caption, func[, options[, runner]])

• caption: <string> name of the test
• func: <Function> test function
  • test: <ImperativeTest> test instance
• options: <TestOptions>
  • run: <boolean> auto start test, default: true
  • async: <boolean> if true do nothing, if false auto-end test on nextTick after func run, default: true
  • timeout: <number> time in milliseconds after which test is considered timeouted.
  • parallelSubtests: <boolean> if true subtests will be run in parallel, otherwise subtests are run sequentially, default: false
  • dependentSubtests: <boolean> if true each subtest will be executed sequentially in order of addition to the parent test short-circuiting if any subtest fails, default: false
• runner: <Runner> runner instance to use to run this test

Returns: <ImperativeTest> test instance

Create a test case.

testSync(caption, func, options = {}, runner = runnerInstance)

Create a synchronous test

Simple wrapper for test() setting async option to false.

testAsync(caption, func, options = {}, runner = runnerInstance)

Create an asynchronous test

Simple wrapper for test() setting async option to true.