test.js

Combinational test framework.

This is not meant as a replacement for more advanced and feature-rich testing frameworks, instead this is a minimalist and complete experimental implementation of a specific approach to testing, i.e. combinational testing

Note, this module is experimental and can change quite allot within a shot to mid timeframe, use at your own risk, though ideas, feedback and suggestions are welcome.

Features

• Simple / minimalist implementation
• Supports combinational as well as unit testing paradigms

Planned

• Multiple modifier chaining
• Replaceable/extensible assertion library

Contents

• test.js
  • Features
    • Planned
  • Contents
  • Architecture
    • Combinational testing
    • Unit testing
  • Installation
  • Basic usage
  • CLI
  • Components
    • DEFAULT_TEST_FILES
    • IGNORE_TEST_FILES
    • Merged
      • <merged>.create(..)
      • <merged>.members
      • <merged>.size / <merged>.usize
      • <merged>.add(..) / <merged>.remove(..)
      • <merged>.clear()
      • <merged>.keys(..) / <merged>.values(..) / <merged>.entries(..)
      • <merged>.toObject(..)
      • <merged>.checkShadowing(..)
      • <merged>.handleShadowing(..)
      • <member>.filename
    • TestSet
    • BASE_TEST_SET
    • Setups(..) / Setup(..) (Merged)
    • Modifiers(..) / Modifier(..) (Merged)
    • Tests(..) / Test(..) (Merged)
    • Cases(..) / Case(..) (Merged)
    • Assert(..)
    • run(..)
  • Advanced components
    • runner(..)
    • parser(..)
  • Utilities
    • getCallerFilename()
  • License

Architecture

This package implements two testing schemes:

• Combinational testing
  Here the user sets up a set of Setup, Modifier and Test functions and the system chains different combinations of the three and runs them.
• Unit testing
  Simple independent tests.

Combinational testing

In general the idea here is that you define three things:

• Setups that build a test context and objects (the setup),
• Modifiers that modify the setup in some way,
• Tests that test some aspect of a setup.

The system builds chains in the form:

setup -> modifier* -> test

Where modifier can me either single or a chain of modifiers.

A setup and modifier can also include assertions/tests for direct testing and sanity checks.

The system defines a blank pass-through modifier and test, to alleviate the requirement on the user to define at least one modifier and test, so in cases where it makes no sense only a setup is required.

Note that each element can reference and re-use other elements so for example a modifier can call (re-use) other modifiers to avoid code duplication.

This makes it simple to define procedural/generative tests.

Unit testing

This is the traditional self-contained test approach.

Installation

$ npm install -i ig-test

And to install the global CLI interface

$ npm install -g ig-test

Basic usage

Create a test script

$ touch test.js
$ chmod +x test.js

Note that the test script should be named either "test.js" or "<something>-test.js" for the system to find it automatically.

The code:

#!/usr/bin/env node

var test = require('ig-test')

// XXX add assert examples...

test.Setups({
    state: function(assert){
        return {
            a: 123,
            b: 321,
        } },
})

test.Modifiers({
    inc: function(assert, state){
        Object.keys(state)
            .forEach(function(k){
                state[k] += 1 })
        return state },    
})

test.Tests({
    
})

test.Cases({
    
})

// make the test runnable as a standalone script...
__filename == (require.main || {}).filename
    && tests.run()

Run the tests

$ node ./test.js

or

$ runtests

CLI

$ npm install -g ig-test

Basic help

$ runtests --help 
Usage: test.js [OPTIONS] [CHAIN] ...

Run tests.

Tests run by test.js can be specified in one of the 
following formats:

        <case>
        <setup>:<test>
        <setup>:<modifier>:<test>

Each of the items in the test spec can be a "*" indicating
that all relevant items should be used, for example:

        $ ./test.js basic:*:*

Here test.js is instructed to run all tests and modifiers
only on the basic setup.

Zero or more sets of tests can be specified.

When no tests specified test.js will run all tests.

Options:
        -h,  --help             - print this message and exit
        -v,  --version          - show test.js verion and exit
        -l,  --list=PATH        - list available tests;
                                  note that if passing files via -f explicitly they
                                  must precede the -l/-list flag;
                                  this has the same defaults as -f
             --list-found=PATH  - like -list but print found test modules and exit
        -f,  --test-file=PATH   - test script or filename pattern, supports glob;
                                  this flag can be given multiple times for
                                  multiple paths/patterns
                                  (default: **/?(*-)test.js)
        -i,  --ignore=PATH      - path/pattern to ignore in test file search
                                  (default: node_modules/**)
             --verbose          - verbose mode
                                  (env: $VERBOSE)

Examples:
        $ ./test.js             - run all tests.
        $ ./test.js basic:*:*   - run all tests and modifiers on "basic" setup.
                                  (see test.js -l for more info)
        $ ./test.js -v example  - run "example" test in verbose mode.
        $ ./test.js native:gen3:methods init:gen3:methods
                                - run two tests/patterns.

List available test components

$ runtests --list 

XXX chains

XXX notes on coverage

Components

DEFAULT_TEST_FILES

glob pattern(s) used to find test files by default.

DEFAULT_TEST_FILES =
    undefined
    | <path>
    | [ <path>, .. ]

Default value: "**/?(*-)test.js"

IGNORE_TEST_FILES

A list of glob patterns to ignore while searching for tests.

IGNORE_TEST_FILES =
    undefined
    | [ <path>, .. ]

Default value: ['node_modules/**']

Merged

Implements a merged collection of instances (members).

Create a new collection:

Merged.create(<name>)
    -> <merged>

Add members to collection:

<merged>({ <key>: <func>, .. })
    -> <member>

<merged>(<key>, <func>)
    -> <member>

On construction this will assign the input object / <key>-<func> into the resulting <member>/instance object.

Each <member>/instance created is added to the constructor as a member (i.e. added into .members)

Provides a set of methods and properties to access/introspect the merged (hence the name) attributes of the members (i.e. .keys(..), .values(..), .entries(..), .size/.usize and .members).

Note that though Merged itself is a collection, it is not designed to be used directly as a collection, use it to create new collections or as a prototype for inheritance.

<merged>.create(..)

Create a new Merged collection (member constructor).

Merged.create(<name>)
    -> <merged>

<merged>.members

List of members / instances of Merged in order of creation.

<merged>.size / <merged>.usize

Number of members including the "-" members and not including respectively.

<merged>.add(..) / <merged>.remove(..)

Add / remove a member.

<merged>.add(<member>)
    -> <merged>

<merged>.remove(<member>)
    -> <merged>

<merged>.clear()

Remove (clear) all the members.

<merged>.keys(..) / <merged>.values(..) / <merged>.entries(..)

<merged>.keys()
<merged>.keys(<merged>)
    -> <list>

<merged>.values()
<merged>.values(<merged>)
    -> <list>

<merged>.entries()
<merged>.entries(<merged>)
    -> <list>

These are similar to Object.keys(..) / Object.values(..) / Object.entries(..) but will also if called without arguments return a list of the callers member keys/values/entries respectively.

Note that members' attributes can shadow previous member attributes, only one value per key will be returned. <merged> will warn when adding a member of its attributes will shadow already existing members' attributes (see: <merged>.checkShadowing(..) and <merged>.handleShadowing(..));
Also note that the check for shadowing is performed when the <member> is created and not when new attributes are added manually.

<merged>.toObject(..)

Create an object containing all visible member attributes.

<merged>.toObject()
    -> <object>

<merged>.checkShadowing(..)

Find all shadowed attributes within <merged>.

<merged>.checkShadowing()

Find all attributes in <merged> that will be shadowed by <member>

<merged>.checkShadowing(<member>)
    -> <list>

<merged>.handleShadowing(..)

Will be called on <member> construction when attribute shadowing is detected.

`<merged>.handleShadowing(<attr>)`
    -> <merged>

By default this will print a warning and continue, but can be overloaded by the user to react to shadowing in a different manner.

<member>.filename

The filename where the <member> was defined.

TestSet

XXX

BASE_TEST_SET

XXX

Setups(..) / Setup(..) (Merged)

XXX

A subclass or rather sub-constructor of Merged.

Note that Setups and Setup are references to the same object, they exists for better readability in cases when we add a single element (<key>-<func> pair) or a bunch of elements (object), for example:

// single element...
test.Setup('some-setup', 
    function(){
        // ...
    })

// arbitrary number of elements...
test.Setups({
    'some-other-setup': function(){
        // ...
    },

    // ...
})

Modifiers(..) / Modifier(..) (Merged)

XXX

A sub-constructor of Merged.

Tests(..) / Test(..) (Merged)

XXX

A sub-constructor of Merged.

Cases(..) / Case(..) (Merged)

XXX

A sub-constructor of Merged.

Assert(..)

XXX this may still change...

run(..)

Run the test system.

run()
run(<tests>)
run(<default-files>)
run(<default_files>, <tests>)
    -> <parse-result>

This will:

• parse process.argv
• locate and run tests
• report basic stats

<tests> format:

{
    setups: <stups>,
    modifiers: <modifiers>,
    tests: <tests>,
    cases: <cases>,
}

Advanced components

runner(..)

The default test combinator and runner.

parser(..)

The default ig-argv parser setup.

Utilities

getCallerFilename()

Returns the filename of the module where getCallerFilename() is called.

License

BSD 3-Clause License

Copyright (c) 2016-2020, Alex A. Naanou,
All rights reserved.