impTest

• Overview
• Installation
• Test Project Configuration
  • Project Configuration Generation
  • GitHub Credentials Configuration
  • Environment Variables
• Writing Tests
  • Tests for Bi-directional Device-Agent Communication
  • Asynchronous Testing
  • Builder Language
  • External Commands
  • Assertions
  • Diagnostic Messages
  • A Test Case Example
• Running Tests
  • Selective Test Runs
  • Debug Mode
• For impTest Tools Developers
• License

Overview

impTest is a set of tools intended to run unit tests that are built with the impUnit test framework. impTest leverages Electric Imp’s Build API to deploy and run the code on imp-enabled devices. All of the tools are written in Node.js and are fully open source.

impTest makes use of the following files and directories:

Test Project is the directory (with all subdirectories) where the tests are located.

There is one Test Project Configuration file per Test Project directory. Test Project Configuration contains all of the settings related to all of the tests used in Test Project.

Project Home is the directory where Test Project Configuration is located.

All files located in Project Home (and in its subdirectories) are considered as files with Test Cases if their names match the patterns specified in Test Project Configuration.

Test Case is a class inherited from the ImpTestCase class. There can be several Test Cases (classes) in a file. A Test Case (class) can contain several tests (methods), each of which should be prefixed test, eg. testEverythingOk().

In order to work with impTest you need to:

• Install impTest
• Create or Update Test Project Configuration
• Write or Update Tests
• Run Tests

If you want to update impTest itself, please see For impTest Tools Developers.

Installation

Node.js 4.0 or greater is required. You can download the Node.js pre-built binary for your platform or install Node.js via package manager.

Once node and npm are installed, you must execute the following command to set up impTest:

npm i -g imptest

Test Project Configuration

A configuration file is a JSON file that contains the following key-value pairs:

| Key | Description | | --- | --- | | apiKey | Your Build API key provides access to Build API. For security reasons we strongly recommend that you define the Build API key as an environment variable | | devices | A set of Device IDs that specify the devices that must be used for tests execution | | modelId | The ID of the Model that the devices are assigned to | | deviceFile | A path to a file with the device source code that is deployed along with the tests. false is used if no additional code is needed | | agentFile | A path to a file with the agent source code that is deployed along with the tests. false is used if no additional code is needed | | tests | A set of patterns that impTest uses to search for files with Test Cases. If ** is alone in the path portion, then it matches zero or more directories and subdirectories that need to be searched. It does not crawl symlinked directories. The pattern default value is ["*.test.nut", "tests/**/*.test.nut"]. Do not change this value if there is a plan to run agent and device test code together | | stopOnFailure | Set this option to true if you want to stop an execution after a test failure. The default value is false | | builderCache | Set this option to true if you want to enable the builder cache for remote libraries. The default value is false | | allowDisconnect | Set this option to true if you want the test sessions to stay alive on temporary device disconnect. The default value is false | | timeout | A timeout period (in seconds) after which the tests are considered as failed. Asynchronous tests are interrupted. The default value is ten seconds |

This is the format of the configuration file, though the settings can be listed in any order:

{ "apiKey":          <string>,                 // Build API key, optional
  "modelId":         <string>,                 // Model ID
  "devices":         <string array>,           // Device IDs
  "deviceFile":      <string or false>,        // Device code file. Default: "device.nut"
  "agentFile":       <string or false>,        // Agent code file. Default: "agent.nut"
  "tests":           <string or string array>, // Test file search pattern. Default: ["*.test.nut", "tests/**/*.test.nut"]
  "builderCache":    <boolean>,                // Enable build cache? Default: false
  "stopOnFailure":   <boolean>,                // Stop tests execution on failure? Default: false
  "allowDisconnect": <boolean>,                // Keep the session alive on device disconnects? Default: false
  "timeout":         <number>                  // Async test methods timeout, seconds. Default: 10
}

Project Configuration Generation

The Test Project Configuration file can be created or updated by the following command:

imptest init [-c <configuration_file>] [-d] [-f]

where:

• -d — prints the debug output.
• -c — provides a path to the configuration file. A relative or absolute path can be used. Generation fails if any intermediate directory in the path does not exist. If the -c option is not specified, the .imptest file in the current directory is assumed.
• -f — updates (overwrites) an existing configuration. If the specified configuration file already exists, this option must be explicitly specified to update the file.

During the command execution you will be asked for configuration settings in either of the following cases:

• if a new Test Project Configuration is being created, the default values of the settings are offered;
• if the existing Test Project Configuration is being updated, the settings from the existing configuration file are offered as defaults.

GitHub Credentials Configuration

Sources from GitHub can be included in test files.

For unauthenticated requests, the GitHub API allows you to make up to 60 requests per hour. To overcome this limitation, you can provide user credentials.

For security reasons, we strongly recommend that you provide the credentials via Environment Variables. However, there is also a way to store the credentials in a special file (one file per Test Project). The file can be created or updated by the following command:

imptest github [-g <credentials_file>] [-d] [-f]

where:

• -d — prints the debug output/
• -g — provides a path to the file with GitHub credentials. A relative or absolute path can be used. Generation fails if any intermediate directory in the path does not exist. If -c option is not specified, the .imptest-auth file in the current directory is assumed.
• -f — updates (overwrites) an existing file. If the specified file already exists, this option must be explicitly specified to update it.

The file syntax is as follows:

{ "github-user": "user",
  "github-token": "password_or_token" }

Environment Variables

For security reasons, we strongly recommend that you define your Build API key and GitHub credentials as environment variables, as follows:

• apiKey -> IMP_BUILD_API_KEY — to deploy and run the code on imp devices via Electric Imp Build API.
• github-user -> GITHUB_USER — to include external sources from GitHub.
• github-token -> GITHUB_TOKEN — to include external sources from GitHub.

Writing Tests

The following are the basic steps you need to follow in order to write tests:

• Choose the name and location of your file with tests. You can have several files with tests in the same or different locations.
  • The name with the path, relative to Project Home, must conform to the patterns specified in the Test Project Configuration file within your Test Project.
  • The file is treated as agent test code if agent is present in the file name. Otherwise the file is treated as device test code.
  • By default, the test code runs either on device or agent. If your Test Case must run on both the device and its agent, there is a way that allows you to execute agent and device test code together.
• Add a Test Case class that inherits from the ImpTestCase class. A file can have several Test Cases. Test Cases can have identical names if they are in different files.
• Add and implement tests: methods whose names start with test. Every Test Case can have several tests.
• Additionally, any Test Case can have setUp() and tearDown() methods to perform the environment setup before the tests execute and then perform cleaning-up afterwards.

A test method can be designed as synchronous (by default) or asynchronous.

A test file must not contain any #require statement. Instead, an include from GitHub must be used. For example: #require "messagemanager.class.nut:1.0.2" must be replaced with @include "github:electricimp/MessageManager/[email protected]".

Here is an example of a simple Test Case:

class MyTestCase extends ImpTestCase {
    function testAssertTrue() {
        this.assertTrue(true);
    }

    function testAssertEqual() {
        this.assertEqual(1000 * 0.01, 100 * 0.1);
    }
}

Tests for Bi-directional Device-Agent Communication

To test interaction between a device and an agent, impTest allows developers to extend tests with corresponding logic implemented on the other side of the link (agent or device, respectively). The test “extensions” can be used to emulate real device-agent interaction and communication.

To identify partners file uniquely, there are some restrictions imposed on the test extensions:

The Test case class must be located either in the device code or the agent code, but not in both. Whichever of the two it is included in, that is the TestFile — the other file is the PartnerFile. Both of these files must be located in the same directory on the disk.

TestFile and PartnerFile must be named as follows: TestName.(agent|device)[.test].nut, ie. they need to have the same TestName prefix and the same .nut suffix.

TestFile is indicated by the .test string in its filename; PartnerFile must not feature this string in the suffix.

The type of execution environment is indicated by either .device or .agent in the file name — if TestFile contains .agent, PartnerFile must have .device, and vice versa.

For example, "Test1.agent.test.nut" (test file) and "Test1.device.nut" (partner file).

Due to partner special naming do not change the default value of "Test file search pattern".

Further examples of test extensions can be found at sample7.

Asynchronous Testing

Every test method (as well as setUp() and tearDown()) can be either synchronous (the default) or asynchronous.

Methods must return an instance of Promise to notify that it needs to do some work asynchronously.

The resolution of the Promise indicates that all test have been passed successfully. The rejection of a Promise denotes a failure.

Example

function testSomethingAsynchronously() {
    return Promise(function (resolve, reject){
        resolve("It's all good, man!");
    });
}

Builder Language

Builder is supported by impTest. The Builder language combines a preprocessor with an expression language and advanced imports.

Example

@set assertText = "Failed to assert that values are"

this.assertEqual(
    expected,
    actual,
        "@{assertText}"
        + " equal in '@{__FILE__}'"
        + " at line @{__LINE__}"
    );

__FILE__ and __LINE__ variables are defined in the Builder, they can be useful for debugging information. Below is a usage example:

this.assertEqual(
    expected,
    actual,
    "Failed to assert that values are"
        + " equal in '@{__FILE__}'"
        + " at line @{__LINE__}"
);

It is possible to define and propagate custom variables through a separate configuration file which syntax is similar to the github credential file:

{ "pollServer": "http://example.com",
  "expectedAnswer": "data ready" }

The default file name is .imptest-builder but an alternative name can be selected with the -b <builder_config_file> command line option as follows:

imptest test -b tests/test1/.test1-builder-config

Now Builder will be able to process any custom variables encountered in the source code.

local response = http.get("@{pollServer}", {}).sendsync();
this.assertEqual(
    "@{expectedAnswer}",
    response,
    "Failed to get expected answer"
);

External Commands

A test can call a host operating system command as follows:

// Within the test case/method
this.runCommand("echo 123");
// The host operating system command `echo 123` is executed

If the execution timeout of an external command expires (the timeout is specified by the timeout parameter in the Test Project Configuration file) or exits with a status code other than 0, the test session fails.

Assertions

The following assertions are available in tests:

assertTrue()

this.assertTrue(condition, [message]);

Asserts that the condition is truthful.

Example

// OK
this.assertTrue(1 == 1);

// Fails
this.assertTrue(1 == 2);

assertEqual()

this.assertEqual(expected, actual, [message])

Asserts that two values are equal.

Example

// OK
this.assertEqual(1000 * 0.01, 100 * 0.1);

// Failure: Expected value: 1, got: 2
this.assertEqual(1, 2);

assertGreater()

this.assertGreater(actual, cmp, [message])

Asserts that a value is greater than some other value.

Example

// OK
this.assertGreater(1, 0);

// Failure: Failed to assert that 1 > 2
this.assertGreater(1, 2);

assertLess()

this.assertLess(actual, cmp, [message])

Asserts that a value is less than some other value.

Example

// OK
this.assertLess(0, 1);

// Failure: Failed to assert that 2 < 2
this.assertLess(2, 2);

assertClose()

this.assertClose(expected, actual, maxDiff, [message])

Asserts that a value is within some tolerance from an expected value.

Example

// OK
this.assertClose(10, 9, 2);

// Failure: Expected value: 10пїЅ0.5, got: 9
this.assertClose(10, 9, 0.5);

assertDeepEqual()

this.assertDeepEqual(expected, actual, [message])

Performs a deep comparison of tables, arrays and classes.

Example

// OK
this.assertDeepEqual({"a" : { "b" : 1 }}, {"a" : { "b" : 0 }});

// Failure: Missing slot [a.b] in actual value
this.assertDeepEqual({"a" : { "b" : 1 }}, {"a" : { "_b" : 0 }});

// Failure: Extra slot [a.c] in actual value
this.assertDeepEqual({"a" : { "b" : 1 }}, {"a" : { "b" : 1, "c": 2 }});

// Failure: At [a.b]: expected "1", got "0"
this.assertDeepEqual({"a" : { "b" : 1 }}, {"a" : { "b" : 0 }});

assertBetween()

this.assertBetween(actual, from, to, [message])

Asserts that a value belongs to the range from from to to.

Example

// OK
this.assertBetween(10, 9, 11);

// Failure: Expected value in the range of 11..12, got 10
this.assertBetween(10, 11, 12);

assertThrowsError

this.assertThrowsError(func, ctx, [args = []], [message])

Asserts that the func function throws an error when it is called with the args arguments and the ctx context. Returns an error thrown by func.

// OK, returns "abc"
this.assertThrowsError(function (a) {
    throw a;
}, this, ["abc"]);

// Failure: Function was expected to throw an error
this.assertThrowsError(function () {
    // Throw "error";
}, this);

Diagnostic Messages

Return values other than null are displayed in the console when a test succeeds and can be used to output the following diagnostic messages:

Test cases can also output informational messages with:

this.info(<message>);

A log of a failed test looks as follows:

This means that the execution of the testMe() method in the MyTestCase class has failed: the incorrect syntax is in line 6 of the test file (containing the MyTestCase class).

A Test Case Example

The utility file myFile.nut contains the following code:

// (optional) Async version, can also be synchronous

function setUp() {
    return Promise(function (resolve, reject){
        resolve("We're ready");
    }.bindenv(this));
}

The Test Case code is as follows:

class TestCase1 extends ImpTestCase {

@include __PATH__+"/myFile.nut"

    // Sync test method
    function testSomethingSync() {
        this.assertTrue(true);    // OK
        this.assertTrue(false);   // Fails
    }

    // Async test method
    function testSomethingAsync() {
        return Promise(function (resolve, reject){
            // Return in 2 seconds
            imp.wakeup(2 /* 2 seconds */, function() {
                resolve("something useful");
            }.bindenv(this));
        }.bindenv(this));
    }

    // (optional) Teardown method - cleans up after the test
    function tearDown() {
        // Clean-up here
    }
}

Running Tests

Use this command to run the tests:

imptest test [-c <configuration_file>] [-g <credentials_file>] [-b <builder_file>] [--builder-cache=true|false] [-d] [testcase_pattern]

where:

• -c — this option is used to provide a path to the Test Project Configuration file. A relative or absolute path can be used. If the -c option is left out, the .imptest file in the current directory is assumed.
• -g — this option is used to provide a path to file with GitHub credentials. A relative or absolute path can be used. If the -g option is left out, the .imptest-auth file in the current directory is assumed.
• -b — this option is used to provide a path to file with Builder variables. A relative or absolute path can be used. If the -b option is left out, the .imptest-builder file in the current directory is assumed.
• --builder-cache — enable (if =true) / disable (if =false) builder cache for this test run. If not specified, defined by the setting in the test project configuration.
• -d — prints debug output, stores device and agent code.
• testcase_pattern — a pattern for selective test runs.

The impTest tool searches all files that match the filename patterns specified in the Test Project Configuration. The search starts with Project Home. The tool looks for all Test Cases (classes) in the files. All the test methods from those classes are considered as tests for the current Test Project.

The optional testcase_pattern selects a specific test or a set of tests for execution from all the found tests. If testcase_pattern is not specified, all the found tests are selected for execution.

The selected tests are executed in an arbitrary order.

Every test is treated as failed if an error has been thrown. Otherwise the test is treated as passed.

Selective Test Runs

The optional testcase_pattern allows you to execute a single test or a set of tests from one or several Test Cases. The syntax of the pattern is as follows: [testFileName]:[testClass].[testMethod]

where:

• testFileName — the name of the Test Case file. A pattern to filter required files among all conforming to the test file search pattern.
• testClass — the name of the Test Case class. Note Test Cases with identical names can exist in different files that belong to the same Test Project, all of them must be selected.
• testMethod — a test method name

Example

The test file TestFile1.test.nut contains:

class MyTestClass extends ImpTestCase {
    function testMe() {...}
    function testMe_1() {...}
}

class MyTestClass_1 extends ImpTestCase {
    function testMe() {...}
    function testMe_1() {...}
}

The test file TestFile2.test.nut contains:

class MyTestClass extends ImpTestCase {
    function testMe() {...}
    function testMe_1() {...}
}

In this case:

• imptest test TestFile1:MyTestClass.testMe runs the testMe() method in the MyTestClass class of the TestFile1.test.nut file.
• imptest test :MyTestClass.testMe runs the testMe() method in the MyTestClass class from the TestFile1 and TestFile2.test.nut file.
• imptest test :MyTestClass_1 runs all test methods from the MyTestClass_1 class of the first file since it is the only file with the required class.
• imptest test TestFile2 runs all test methods from the TestFile2.test.nut file.
• imptest test :.testMe_1 runs the testMe_1() methods in all classes of all files.

Note Search patterns are allowed for test file names only. A test class and a test method must be fully qualified.

Note If no colon is present in the testcase filter, it is assumed that only the pattern for a file name is specified.

Note An internal class can play the role of a test case. To denote this use case, put "." at the end of the filter. For example, "imptest test :Inner.TestClass." executes all test methods from the Inner.TestClass class.

Builder Cache

Builder cache is intended to improve the build time and reduce the number of requests to external resources. It is only possible to cache external libraries. Builder stores the cache in the .builer-cache folder for up to 24 hours.

Caching is disabled by default. It can be activated during a test project configuration generation.

It is possible to specify whether the builder cache should be enabled or disabled for a given test run. For example, to disable the cache:

imptest test --builder-cache=false

There is no special impTest option or command to remove the builder cache, but you can do this manually:

rm -rf .builder-cache

Debug Mode

The -d option is used to run tests in the debug mode:

• A debug output is switched on. JSON is used to communicate between impUnit test framework and impTest. Communication messages are printed.
• Device and agent code are stored in the ./build folder that is created in Project Home.

The debug mode is useful for analyzing failures.

The following is an example of a debug log:

For impTest Tools Developers

License

The code in this repository is licensed under MIT License.