cymetrics
v1.0.3
Published
Cymetrics is a Cypress post process that looks at durations on an individual test case level and at a file level, to determine the fastest completion time for the amount of test runners available. The average duration of multiple runs will be used, so the
Downloads
3
Maintainers
Readme
Table of Contents
INTRODUCTION
Cymetrics is a Cypress post process that looks at durations on an individual test case level and at a file level, to determine the fastest completion time for the amount of test runners available. The average duration of multiple runs will be used, so the more cymetrics is used, the more accurate it becomes.
Check out the scenario section for a better understanding of cymetrics, or the sample project, cypress-playground demonstrating cymetrics' use.
Example
Scenario
The Cypress test suite contains 4 files and each file contains 2 test cases. We normally run this suite nightly on a single Jenkins runner. We recently got a second Jenkins test runner and would like to as evenly as possible distribute the tests between our 2 test runners in order to get the shortest overall test duration.
After one run of the entire test suite we get the following information:
- file_1.js: 10 minutes
- tc_1.1:......2 minutes
- tc_1.2:......8 minutes
- file_2.js: 2 minutes
- tc_2.1.......1 minute
- tc_2.2.......1 minute
- file_3.js: 15 minutes
- tc_3.1:......3 minutes
- tc_3.2:......12 minutes
- file_4.js: 9 minutes
- tc_4.1:......4 minutes
- tc_4.2:......5 minutes
Total Runtime WITHOUT cymetrics: 36 minutes
Cymetrics will now take this information and find the shortest overall duration possible given 2 test runners.
Cymetric Test Case Level Output
Scenario output at the test case level: (Total Runtime WITH cymetrics: 18 minutes)
- Test Runner 1
- tc_3.2.......12 minutes
- tc_4.1.......4 minutes
- tc_1.1.......2 minutes
- Test Runner 2
- tc_1.2.......8 minutes
- tc_4.2.......5 minutes
- tc_3.1.......3 minutes
- tc_2.1.......1 minute
- tc_2.2.......1 minute
Real output would look something like this:
{
"g1": {
"files": [
"tc_3.2",
"tc_4.1",
"tc_1.1"
],
"estTotalDuration": 1080000
},
"g2": {
"files": [
"tc_1.2",
"tc_4.2",
"tc_3.1",
"tc_2.1",
"tc_2.2"
],
"estTotalDuration": 1080000
}
}
Cymetric File Level Output
Scenario output at the file level: (Total Runtime WITH cymetrics: 19 minutes)
- Test Runner 1
- file_3.js...... 15 minutes
- file_2.js.... 2 minutes
- Test Runner 2
- file_1.js.....10 minutes
- file_4.js.......9 minutes
Real output would look something like this:
{
"g1": {
"title": [
"cypress/integration/file_3.js",
"cypress/integration/file_2.js"
],
"estTotalDuration": 1080000,
"command": "npm run spec 'cypress/integration/file_3.js','cypress/integration/file_2.js',"
},
"g2": {
"title": [
"cypress/integration/file_1.js",
"cypress/integration/file_4.js"
],
"estTotalDuration": 1080000,
"command": "npm run spec 'cypress/integration/file_1.js','cypress/integration/file_4.js',"
}
}
REQUIREMENTS
Cymetrics requires Mochawesome reporting as input to produce the final metrics.
In order to use mochawesome correctly, we will add async and
child_process to the /plugins/index.js
file.
These are optional and feel free to stray from these packages.
INSTALLATION
The following installations assumes you have met the requirements within the requirements section.
Async
npm install async --save-dev
Mochawesome
npm install mocha --save-dev
npm install cypress-multi-reporters --save-dev
npm install mochawesome --save-dev
npm install mochawesome-merge --save-dev
npm install mochawesome-report-generator --save-dev
Set the mochawesome reporter settings within the cypress.json
file
{
"reporter": "mochawesome",
"reporterOptions": {
"reportDir": "cypress/reports/testCaseResults",
"overwrite": false,
"html": false,
"json": true
}
}
Add these lines to the package.json
in the script section.
{
"scripts": {
"clean-reports": "npx rimraf cypress/reports/\"**\"",
"posttest": "npm run mocha-merge && npm run mocha-marge",
"mocha-merge": "npx mochawesome-merge 'cypress/reports/testCaseResults/*json' > cypress/reports/mochawesomeMerged.json",
"mocha-marge": "npx marge cypress/reports/mochawesomeMerged.json -f report -o cypress/reports/html"
}
}
Within the plugins/index.js
call the mochawesome scripts
on('before:run', (config) => {
series([() => exec('npm run clean-reports')]);
})
on('after:run', async (config) => {
series([() => exec('npm run posttest')]);
})
Cymetrics
npm install cymetrics
Set the mochawesome reporter settings within the cypress.json
file
{
"load_balancer": {
"outputDirectory": "output",
"testCaseOutputFileName": "tcDataOutput",
"fileOutputFileName": "fileDataOutput",
"balancedFilesOutputFileName": "balancedFiles",
"balancedTcOutputFileName": "balancedTestCases",
"testRunnerCount": "2",
"mochawesomeReport": "cypress/reports/mochawesomeMerged.json"
}
}
Within the plugins/index.js
call cymetrics in an after:run
block
on('after:run', async (config) => {
await cymetrics.balance(config)
})
Note: the final plugins/index.js
file should look like this:
const cymetrics = require('cymetrics')
const series = require('async').series
const {exec} = require('child_process');
module.exports = (on, config) => {
on('before:run', (config) => {
series([() => exec('npm run clean-reports')]);
})
on('after:run', async (config) => {
series([() => exec('npm run posttest')]);
await cymetrics.balance(config)
})
}
Options
outputDirectory - Folder location where cymetrics output will placed
testCaseOutputFileName - File name for unbalanced data on all the test cases
fileOutputFileName - File name for unbalanced data on all files with test cases
balancedFilesOutputFileName - File name for balanced data for all files with test cases
balancedTcOutputFileName - File name for balanced data on all the test cases
testRunnerCount - The amount of 'baskets' to split up test cases and test files
mochawesomeReport - Location of mochawesome file with combined results