@b08/test-runner, seeded from @b08/library-seed, library type: support

This test runner was inspired by several "node-tap" principles of how testing framework should work. Those principles proven to be game changing. Not all of them of course, the rest of node-tap principles are crap.

Principles

Test files should be "normal" programs that can be run directly.

Test framework should not monkey-patch global scope. I used Mocha a lot, and this "run-directly" principle is a great improvement.

Assertions should not throw.

This means that test, containing several assertions, should execute all of them, not stop at the first failed one. This is useful if assertions are independent.

Tests should be independent, to allow running in parallel.

This is no principle for the framework, it is a requirement for the test writer. This is one example of what kind of crap you can find in node-tap readme.

Tests should run as fast as possible.

Here is where node-tap sucks balls. This was the reason for me to write my own runner. 44 tests are taking over 30 seconds on fast PC. Gitlab CI runner fails with timeout while trying to run those tests. Setting a number of jobs does not significantly improve it.

Should I mention my runner takes only 50ms? It does a lot less stuff of course. Only what is actually essential for testing. No fancy output with code fragments. When test has failed I will need to open that file anyway, fancy output or not. That might be not the reason why node-tap is slow as hell, but still.
Link to the actual file for VSCode to open would be much more useful than that crap. Not sure if it is at all possible. Might be implemented later if I find a way.

CLI and programmatic interfaces should be equivalent

Last but not least. It should be easy to integrate the runner into any workflow, gulp or otherwise. This is my own principle, not from node-tap. I had a lot of issues with mocha, and even after it started actually working for me, I had to use old version of gulp-mocha and live with audit vulnerabilities.

Writing tests

There are two ways to write the tests. Both of those will allow to just run any test file with "node build/testSrc/add.spec.js". Also works with original ts if you run it with ts-node.

Spec style:

import { describe } from "@b08/test-runner";

describe("add", it => {
  it("should return sum of two numbers", assert => {
    assert.equal(add(2,3), 5);
  });
});

Tap style:

import { test } from "@b08/test-runner";

test("add should return sum of two numbers", assert => {
  assert.equal(add(2,3), 5);
});

Unlike node-tap, no assertion outside of a test. Also unlike tap, test is a measurable unit. Node-tap counts "suites", i.e. files with tests, which makes little sense.
I think you should be able to move any test to separate file and nothing should change. I.e. tests should be independent units.
"Suite" approach, in my opinion, encourages developer to write co-dependent tests.

Using the runner programmatically

import { runTests } from "@b08/test-runner";

const testOptions = {
  files: "./build/testSrc/**/*.spec.js",
  reporter: "red",
  threads: 4, // more on thread count below 
  testTimeout: 2000, // test will fail if executes longer
  timeout: 30000 // thread will throw if runs longer than that
}

async function gulpTask() {
  await runTests(testOptions);
}

Reporters

Available options:
"verbose" - prints tests, up to 5 green assertions and all red assertions
"brief" - prints tests and red assertions
"red" - prints only red tests and assertions
"summary" - prints test summary
"silent" - self explanatory
If unspecified, defaults to ["verbose", "summary"]
Also can specify custom reporter implementing IReporter interface

Threads

All test files are split into groups, first group is executed in current thread, the rest are executed each in its own worker thread. If thread count is not specified or 0, it will be calculated from number of test files. Up to 19 files will result in one thread, i.e. no worker threads will be created. Up to 29 files - 2 threads and so on, plus one thread for each 10 extra files up to logical cpu's count in the system. Be aware that worker thread creation is expensive, so if your tests take less than 100-200ms, thread creation might make it slower. For most cases one thread is more than enough.

Using the runner via CLI

npx gulp test-runner "./build/**/*.spec.js" --reporter=minimal --reporter=summary --timeout 30000

Limitation - running only via npx is supported, do not try and install this package globally.