attester
v4.1.0
Published
Command line tool to run Javascript tests in several web browsers.
Downloads
169
Readme
Attester
attester is a command line tool allowing to run JavaScript tests in multiple web browsers at once.
It starts an internal web server, then starts a set of web browsers, makes them execute the tests, and finally writes test reports.
It is written in JavaScript, to be run with node.js.
Features
- Supports multiple browser instances to run tests in parallel
- Includes instrumentation for code coverage with node-coverage
- Supports PhantomJS for fully headless tests
- Compatible with most other web browsers
- Test results output formats:
- Json file
- JUnit-style single XML file
- JUnit-style set of files, format accepted by Sonar
- Code coverage output formats:
- node-coverage json file
- lcov file, accepted by Sonar (currently only for line coverage)
- Supports Aria Templates unit tests and Mocha with either Chai or Expect.js.
- Adding support for other test frameworks is as simple as adding an adapter for that test framework.
License
Dependencies & installation
You'll need node.js. Attester is available in npm repository; you can use npm install attester
in your console, or clone this repository and then issue npm install
.
If you want to make use of PhantomJS headless testing, you'll additionally need to install PhantomJS and make sure it's in your PATH
.
Installing PhantomJS on Windows
- Download phantomjs-1.9.8-windows.zip and extract it.
- Move the contents of
phantomjs-1.9.8-windows
toC:\bin\phantomjs
- Add
C:\bin\phantomjs
toPATH
- Check that it works by issuing
phantomjs --version
incmd
Installing PhantomJS on Ubuntu
Quick setup on 64bit Ubuntu:
cd /usr/local/share/
sudo wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2
sudo tar jxvf phantomjs-1.9.8-linux-x86_64.tar.bz2
sudo ln -s /usr/local/share/phantomjs-1.9.8-linux-x86_64/ /usr/local/share/phantomjs
sudo ln -s /usr/local/share/phantomjs/bin/phantomjs /usr/local/bin/phantomjs
If you have 32bit version, replace x86_64
with i686
in the commands above.
Check that it works by issuing phantomjs --version
in console.
Usage
attester [<options>] [<configuration file>]
Configuration file
The configuration file describes the test campaign to execute. It can be either in the YAML
(with a .yml
or .yaml
extension) or in the JSON format (with a .json
extension).
Here is a sample .yml
configuration file with explanations about each setting:
resources: # Specifies which files will be accessible through the web server:
'/': # This configures resources for the root of the web server:
- 'src/main/js' # It is possible to specify one or more directories for each path.
- 'src/tests' # When requesting a file, the first directory which contains it is used.
'/aria': # This configures resources for the /aria path on the server:
- 'libraries/aria'
tests:
# Describes test configuration for each type of test
# Only Aria Templates tests are supported currently, but other types of tests will be added in the future
aria-templates:
# There are two ways to specify which tests have to be executed:
# either by their classpath or by their file path
# It is possible to combine both.
# With file paths, it is possible to use patterns.
classpaths:
includes:
- MainTestSuite
excludes:
- test.sample.MyUnfinishedTest
browserExcludes: # Allows to specify some excluded classpaths specific to each browser
PhantomJS: # Each browser listed in browserExcludes must exactly match the name of a browser in the browsers section
- test.sample.MyTestFailingInPhantomJS
files:
rootDirectory: 'src/tests'
includes:
- 'test/example/*TestCase.js'
excludes:
- 'test/example/*SpecialTestCase.js'
browserExcludes: # Allows to specify some exclusion patterns specific to each browser
PhantomJS: # Each browser listed in browserExcludes must exactly match the name of a browser in the browsers section
- 'test/example/*NoPhantomTestCase.js'
bootstrap : '/aria/bootstrap.js' # Path to the bootstrap file of Aria Templates. This is the default value.
extraScripts: # Path to extra scripts to be inserted in the test page just after the Aria Templates bootstrap file.
- '/test/testEnvConfig.js'
rootFolderPath : '/' # Root folder path (Aria.rootFolderPath variable) This is the default value.
debug : true # Enables or disables Aria Templates debug mode (Aria.debug variable). This is the default value.
memCheckMode : true # Enables or disables memory check mode (Aria.memCheckMode variable). This is the default value.
mocha:
files:
includes:
- 'test/example/*.js'
excludes:
- 'test/example/*.conf.js'
browserExcludes: # Allows to specify some exclusion patterns specific to each browser
PhantomJS: # Each browser listed in browserExcludes must exactly match the name of a browser in the browsers section
- 'test/example/notWorkingInPhantomJS/*.js'
ui: 'bdd' # User-interface (bdd|tdd|exports). Defaults to bdd
ignoreLeaks: false # Optional to ignore global variable leaks
globals: # Optional, allow the given list of global [names]
- 'myGlobal'
assertion: 'expect' # Assertion library to be included. It could be either expect or chai or any external url
coverage:
files:
# If the files section is present, the files specified here will be instrumented by attester for code coverage.
rootDirectory: 'src/main/js'
includes:
- '**/*.js'
# Files can be pre-instrumented by node-coverage or the NodeCoverageInstrument visitor of atpackager
# In that case, information file(s) produced by the instrumentation tool should be specified in the infoFiles array:
infoFiles:
- 'coverage-instrumentation.json'
test-reports: # Path for each test report type:
json-file: 'report.json'
xml-file: 'report.xml'
xml-directory: 'reports'
coverage-reports: # Path for each coverage test report type:
json-file: 'coverage.json'
lcov-file: 'coverage.lcov'
browsers:
# If the browsers section is not present, attester will not make any difference between browsers
# and run each test only once (in whatever browser is connected)
# However, if the browsers section is present, each test will be run once in each browser listed here.
# (and browsers not listed here will have nothing to do if they are connected)
- 'PhantomJS'
- 'Opera'
# It's possible to distinguish browsers by operating systems, read more below
- 'Safari on Mac OS X'
- 'Chrome on Windows 7'
- 'Chrome on Ubuntu'
- 'Chrome on Android'
# It is also possible to distinguish several versions of the same browser:
- 'IE 7'
- 'IE 8'
- 'IE 9'
- 'IE 10'
# Browser version can be also a semver-compliant string; see https://github.com/isaacs/node-semver
- 'Firefox 3.6'
- 'Firefox >=20'
# In the logs, browser will be identified using the name, version and OS as provided. If you want to change that, add an alias at the end using 'as':
- 'Chrome 28 as Chrome Stable 28'
- 'Chrome 30 as Chrome Canary 30'
# You can mix all of that options, and have whitespace as you see fit.
- 'Firefox >=25 on Windows 7 as Firefox/Windows'
- 'Firefox >=25 on Ubuntu as Firefox/Ubuntu'
# You can also add flags with the 'with' keyword to specify an environment which cannot be automatically detected.
# Flags have to be explicitly set in the URL when connecting a browser (e.g.: http://localhost:7777/__attester__/slave.html?flags=JAWS).
- 'IE 11 on Windows with JAWS as IE-JAWS'
- 'IE 11 on Windows with mySpecialFlag'
Attester uses ua-parser-js for browser and operating system detection and supports values as returned by ua-parser-js.
Environment Variables
It is possible to build a special portion of the configuration object from an external file using --env
option.
attester --env package.json
This puts the content of package.json
inside the env
section of attester configuration. It is then possible to reference such values with simple templates.
<%= prop.subprop %>
Expands to the value of prop.subprop
in the config. Such templates can only be used inside string values, either in arrays or objects
{
"resources": {
"/": "src"
},
"tests": {
"mocha": {
"files": {
"includes": ["spec/**/*"]
}
}
},
"coverage": {
"files": {
"rootDirectory": "src",
"includes": ["**/*.js"]
}
},
"coverage-reports": {
"json-file": "/reports/coverage-<%= env.version %>.json"
}
}
The configuration above generates a coverage report file called coverage-x.y.z.json
where x.y.z
is the value taken from package.json
or any other file passed referenced by --env
.
Template options can be used both in yaml
and json
file and the environment file can be of any of the two format.
Usual options:
--phantomjs-path <path>
Path to the PhantomJS executable. Attester does not install PhantomJS automatically. The path to the PhantomJS executable can be specified either in the PHANTOMJS_PATH environment variable, or through this command line option. If none are specified, the phantomjs
binary should be in your PATH
for Attester to find it.
--phantomjs-instances <number>
Number of instances of PhantomJS to start (default: 0
).
Additionally, a string auto
can be passed to let the program use the optimal number of instances for best performance (max. 1 per CPU thread).
--run-browser <path>
Path to any browser executable to execute the tests. Can be repeated multiple times to start multiple
browsers or multiple instances of the same browser. Each browser is started with one parameter: the URL to open to start tests.
At the end of the tests, all started processes are killed.
--launcher-config <launcher config>
Path to an attester-launcher configuration file. Browsers configured in this file are automatically started and stopped when needed. Can be repeated multiple times to pass multiple configuration files to attester-launcher.
--env <path>
Path to a yaml
or json
file containing environment options. See section above.
--ignore-errors
When enabled, test errors (not including failures) will not cause this process to return a non-zero value.
--ignore-failures
When enabled, test failures (anticipated errors) will not cause this process to return a non-zero value.
--live-results
When enabled (which is the default, disable with --no-live-results), it is possible to connect to attester
to get live test results. Note that, when enabled, this feature keeps the full log of test results in memory, which increases
the memory consumption of attester.
--host <host>
Host used for the internal web server.
--port <number>
Port used for the internal web server. If set to 0
, an available port is automatically selected.
--public-host <host>
Public host of the internal web server, which, when defined, is used instead of --host to create the URL to be called from browsers.
--public-port <number>
Public port of the internal web server, which, when defined, is used instead of --port to create the URL to be called from browsers.
--server-only
Only starts the web server, and configure it for the test campaign but do not start the campaign. This is useful to
run tests manually.
--max-task-restarts <number>
Maximum number of times a task can be restarted after being interrupted by a browser disconnection (default: 5).
--task-restart-on-failure
When enabled, failing tasks are also automatically restarted as if the browser was disconnected.
This option is especially useful for test campaigns which are not very stable, containing tests which can randomly fail (even if, of course,
having randomly failing tests in a test suite is not recommended!).
--log-level <number>
Level of logging: integer from 0
(no logging) to 5
(trace).
--colors
Uses colors (disable with --no-colors).
--help
Displays a help message and exits.
--version
Displays the version number and exits.
Advanced options
--json-console
When enabled, JSON objects will be sent to stdout
to provide information about tests.
This is used by the junit bridge.
--heartbeats
Delay (in ms) for heartbeats messages sent when --json-console
is enabled. Use 0
to disable them.
--task-timeout
Maximum duration (in ms) for a test task.
--shutdown-on-campaign-end <boolean>
When set to false, will not shut the server down nor exit the process upon finishing the campaign.
Might be useful for debugging.
--predictable-urls <boolean>
If true, resources served by the campaign have predictable URLs (campaign1, campaign2...). Otherwise, the campaign part in the URL is campaign ID.
Configuration file options
Any option configurable through the configuration file can also be specified from the command line with the --config
prefix.
For example, to configure resources, it is possible to use:
attester --config.resources./ path1/root1 --config.resources./ path2/root2 --config.resources./aria path/to/aria
Test Page
It's important to understand exactly what is served by attester to a connected browser.
A slave browser receives a page with a single test and the files necessary to run it.
The test page has access to anything that is listed in resources
.
From the configuration file
tests
is used to build a list of tests for each testing framework
resources
simply defines what is accessible by the page, none of these files is automatically added to the page
Inside test
there are two parameters that deal with files
files
used to generate the list of test filesextraScripts
resources to be included in the test page
It's worth noticing that extraScripts
must be an absolute path from the server root (what is configured in the resources).
Let's see one example
{
"resources" : {
"/" : ["src"],
"/test" : ["spec"]
}
}
The block above configures attester to serve any file inside src
folder from the server root, and files from spec
from /test
.
{
"resources" : {
"/" : ["src"],
"/test" : ["spec"]
},
"tests" : {
"mocha" : {
"files" : {
"includes" : ["spec/**/*.js"]
}
}
}
}
This block tells attester that each JavaScript file in the spec folder is a mocha
test. When dispatching them to connected browsers, each will receive a page containing
- mocha
- an attester adapter to mocha tests
- a single test file
Including source files it's up to the test that can use script loaders like noder, curl, head.js or similar. Using a script loader allows to include in the tested page only what's strictly necessary for the test.
It's also possible to configure attester to include other scripts then the test through extraScripts
.
{
"resources": {
"/": ["src"],
"/test": ["spec"]
},
"tests": {
"mocha": {
"files": {
"includes": ["spec/**/*.js"]
},
"extraScripts": ["/sourceCode.js", "/utilities.js"]
}
}
}
The paths specified in extraScripts
are resources URL. In the example above sourceCode.js
must be physically located inside src
folder.
The files specified in extraScripts
are included in each and every test.