A Testing Problem

When working with selenium and doing it in a way which matters, you drive the browser to a region of the site and then run a bunch of tests. If any of those tests failed you need to run the rest of the tests ignoring the failing test but in such a way which preserves testing integrity.

EX You need to test the cart page. You drive the browser to the cart address and after 3 tests in you start testing the 'clone item' feature. The clone item feature failed. The test broke. You need to move on to test the deleting cart item, and you expect there to be only 1 cart item at start but since the previous test failed that may not be the case.

The immidiate solution is to attempt to clean up a failed test but since you really don't know how the test failed it is impractical. A better solution would be to shut down the system, start over, and restart testing, ignoring all passed and failed tests.

NIMM TEST

Nimm Test uses decorators to setup a testing system similar to what you would find in c# (NUnit or MSTest). Decorators are imported from nimm-test: import {start, end, beforeEach, afterEach, test, fixture, nsFixture, fileFixture} from 'nimm-test';

start

Method decorator will run once at start of fixture

end

Method decorator will run once at end of fixture

nsFixture

Since there's no real namespacing in javascript, each distinct directory name will act as a namespace. all nsFixtures will run once within a directory.

@nsFixture
class CartTests {
    @start
    place_product_in_cart_and_goto_cart() {

    }
    @end
    empty_cart_goback_to_homepage() {

    }
}

fileFixture

File fixtures will run once per file. Thus if a suite of tests were not involved enought to warrant their own directory you can stash them all in a single file.

fixture

All tests are placed into a class decorated by @fixture. Only this fixture accepts beforeEach, afterEach, and test method decorators. But it can also accept start and end decorators (just like fileFixture and nsFixture);

beforeEach

Method will run before each test in a fixture

afterEach

Method will run after each tests in a fixture

test

Only methors marked with this decorators will be tested

Full Method Name

Once the system discovers the tests to run full test name will become a directory path followed by class name, '@', and method name of the test EX 'site/cart/TestCartDelete@shouldDelete'

Running the tests

Once you setup the test structure, here's how to run it. If you want a cli, make your own :D

import {Parser, TestStatus} from 'nimm-test';

(async function() {
    var p = new Parser();
    p.setTests('./myTests', /foo/);
    p.setReporter({
        pipe:function(id, testName, attempt, status, errors) {

        }
    })
    p.setFaultAttempt(3);

    await p.init();

    while(true)
        try {
            await p.startTesting();
            p.quickReport();
            break;
        }
        catch(e) {
            //clean up test system
        }
    }
    
    if (p.tests.some(test=>v.status==TestStatus.failed))
        process.exit(1);
    else 
        process.exit(0);

})()

setTests

Takes 2 arguments. First is a path where all your tests are found, 2nd argument is a regular expression which will run only those tests which full name matches the expression. Pass in null for 2nd argument to run all tests.

setReporter

Takes any number of arguments, each should be a reporter implementing 'pipe' method (IReporter). Pipe will be called at every step of each test with these parameters:

• id of the test
• full test name
• number of current attempts
• status (from TestStatus)
• error array corresponding to each attempt

setFaultAttempt

Set number of attempts a test will try to run before finally failing. A faulting test which ultimately succeeds is marked as StatusEvent.suspect instead of StatusEven.passed. The number defaults to 2 if not set. Meaning, each test will run 2 times, on the 2nd time of failure it will be mared TestStatus.failed.

init

Discover the tests and mark which ones should be run or skipped. Always call init before startTesting.

startTesting

Started testing the system. A failing and faulting test will throw an error, so we wrap it in a while loop. But given that it will not run any passed, failed, suspect, or skipped tests, eventually it will run through the whole system.