TestArmada Manifest

Magellan sends information about system status and test runs using events, that's great, but it's not the most convenient things to report results against. Manifest is a helper class for Magellan that tracks suite runs and builds an easy-to-use DOM style structure with tests, environments, individual test runs and so on.

Creating the SuiteRunResult object

All the information about a Magellan suite-run is stored in a SuiteRunResult object, to create one:

const SuiteRunResult = require("testarmada-manifest");
const srr = new SuiteRunResult();

Connecting to Magellan

There are two entrypoint methods that you need to connect up to Magellan, the first is global messages:

Reporter.prototype.initialize = function (magellanGlobals) {
  const analytics = magellanGlobals.analytics;
  const self = this;
  const deferred = Q.defer();
  deferred.resolve();

  analytics.sync().forEach((message) => srr.globalMessage(message));

  analytics.getEmitter().addListener("message", (message) => {
    srr.globalMessage(message);  
  });

  return deferred.promise;
};

This example is from a Magellan reporter.

The second hookup is onto test messages:

Reporter.prototype.listenTo = function (testRun, test, source) {
  if (test && testRun) {
    source.addListener("message", (event) => {
      srr.testRunMessage(testRun, test, message);
    });
  } else {
    source.addListener("message", (message) => {
      srr.globalMessage(message);  
    });
  }
};

Events from SuiteRunResult

There are some important high level events you should pay attention to.

start and end are sent when the suite run starts and ends. There are no arguments with the message.

testRunStart and testRunEnd is sent when a test run starts and ends. There is one argument which is the testRun object.

newTest is sent when a new test is encountered that was not previously seen. There is only one argument, which is the test object.

There are other events emit for the status of workers, when magellan is idle and busy, and when events are emit from Magellan that the SuiteRunResult object doens't understand. The last one is more for system diagnostics.

Structure of the SuiteRunResult

The SuiteRunResult is a hierarchy:

SuiteRunResult -> Tests -> Environments -> TestRuns

The suite run contains a bunch of tests, each of those is run against different environments, and each environment can be run several times, with a test-run for each time.

SuiteRunResult API

Once you have the events hooked up you'll want to inspect the result of the suite run.

| Property | Description | |-------------------------|-------------------| | tests | The array of test objects | | passed | True if all the tests passed | | failed | True if a single test failed | | retried | The total number of retries | | testRuns | An array of all the tests runs | | timePassing | The total time spent on passing tests | | timeFailing | The total time spent on failing tests | | timeRetrying | The total time spent retrying tests |

There are also high level report properties that you can inspect.

| Property | Description | |-------------------------|-------------------| | testsByNameReport | High level metrics by test name | | testsByEnvironment | An object that contains arrays of test runs organized by environment | | testsByEnvironmentAndTest | An object that returns all of the test runs organized by environment first, then by test name | | testsByEnvironmentReport | High level metrics organized by environment and test name | | reports | Returns a Report object for this suite run (see below) |

Here are some deeper properties you may or may not need.

| Property | Description | |-------------------------|-------------------| | satisfiedWorkers | An array of all the satisfied workers | | satisfiedWorkersTime | Total time spent on satisfied workers | | unsatisfiedWorkers | An array of all the unsatisfied workers | | unsatisfiedWorkersTime | Total time spent on unsatisfied workers |

There are methods associated with tagging tests:

| Property | Description | |-------------------------|-------------------| | setTags(test, tags) | Sets the array of tags for a test with the given name | | testsByTag(tag) | Returns the array of tests for a given tag | | passedByTag(tag) | The total number of tests with that tag that passed | | failedByTag(tag) | The total number of tests with that tag that failed | | timeElapsedByTag(tag) | The total amount of time spent on tests with that tag |

Test API

Each test has these properties:

| Property | Description | |-------------------------|-------------------| | name | The name of the test | | environments | An object with a key for each environment pointing to an Environment object | | testRuns | An array of all of the test runs across all the environments | | passed | True if all the tests passed across all the environments | | retried | The total number of retries across the environments | | timeRetrying | The time spent retrying | | timeElapsed | The total time elapsed across all the environments | | timeForPassed | The total time for all the passed tests, but only for the passing test run |

Environment API

Each environment object has these properties:

| Property | Description | |-------------------------|-------------------| | id | The environment ID | | test | The associated test object | | attempts | The test runs | | retries | The retry test runs | | passingAttempt | The passing attempt, if there was one | | passed | True if this test passed on this environment | | timeRetrying | The time spent retrying | | timeElapsed | The total time elapsed across all the test runs | | timeForPassed | The total time for all the passed tests, but only for the passing test run |

TestRun API

| Property | Description | |-------------------------|-------------------| | test | The associated test object | | environment | The environment object | | status | The status | | passed | True if this test run passed | | metadata | Metadata for the test run | | attemptNumber | The attempt number |

Reports API

| Property | Description | |-------------------------|-------------------| | timeElapsed.top | The top 10 tests sorted by least time elapsed | | timeElapsed.bottom | The bottom 10 tests sorted by least time elapsed | | timeElapsed.complete | All the tests sorted by least time elapsed | | timeElapsed.average | The average time elapsed | | timeElapsed.stddev | The time elapsed standard deviation | | timeForPassed.top | The top 10 tests sorted by least time for the passing test run | | timeForPassed.bottom | The bottom 10 tests sorted by least time for the passing test run | | timeForPassed.complete | All the tests sorted by least time for the passing test run | | timeForPassed.average | The average time for the passing test run | | timeForPassed.stddev | The time for the passing test run standard deviation | | byEnvironment[env].min | The minimum time for the passing test run on the given environment | | byEnvironment[env].max | The maximum time for the passing test run on the given environment | | byEnvironment[env].average | The average time for the passing test run on the given environment | | byEnvironment[env].stddev | The standard deviation for time for the passing test run on the given environment | | byEnvironment[env].avgRetries | The average number of retries for the passing test run on the given environment |

License

Licenses All code not otherwise specified is Copyright Wal-Mart Stores, Inc. Released under the MIT License.