wdio-cucumber-retry-failed-scenarios

A WebdriverIO capability for re-executing Cucumber failed scenarios. This takes care of those random flaky scenarios in the suite which fail during the execution, causing false failures.

Once the automation suite has exectued all the scenarios, JSON report is generated which contains the pass/ fail status of each scenario. This capability takes that report as input and generates a feature file containing all the failed scenarios if the failure threshold is below the threshold specified in the input. The corresponding wdio config file is also generated which will run only the failed scenarios.

Once the failed scenarios have been re-executed, the corresponding execution report can be merged to the initial execution report to give a consolidated view for reporting after updating all those scenarios as passed, after re-execution.

Installation

npm install wdio-cucumber-retry-failed-scenarios

Usage

The Module contains the following functions:

identifyFailedScenarios

const retryFailedScenarios = require('wdio-cucumber-retry-failed-scenarios').retry;

let input = {
    executionResultsFolder: path_to_the_execution_results_folder,
    failureThreshold: an_integer_between_1_to_100,
    configDetails: wdio_configuration_file_details_in_JSON_format
    tag: tag_to_be_applied_to_failure_feature_file (Ex:'@tag')
}

retryFailedScenarios.identifyFailedScenarios(input);

| Parameter | Type | Details | |----------------------------------|--------|-------------------------------------------------------------------------------------------------| | executionResultsFolder | string | Path to folder where the execution results are generated in JSON format. To generate this JSON, use https://www.npmjs.com/package/cucumber-html-reporter | | failureThreshold | int | Threshold percentage of the failures beyond which re-execution will not be triggered and hence failed scenario feature file will not be generated | | configDetails | JSON | wdio configuration file details in JSON format | | tag (optional) | string | Tag expression to be applied at feature level of the failed feature file |

• Returns true if failure percentage in the suite is less than the specified threshold. The failed scenarios would be printed on the console along with generation of the following two files:

  • One Feature file '_failedScenarios.feature' containing all the failed scenarios.
  • WDIO conf file '_failedScenariosConf.js' with 'spec' pointing to the failed feature file.

  The following command can now be used to trigger only the failed scenarios.

  ./node_modules/.bin/wdio _failedScenariosConf.js

  Both the files are generated under the root folder of the project

• Returns false in case failure percentage is 0. Following information would be displayed and no file would be generated:

  No failed scenarios to re-run.

• Returns false in case failure percentage is more than the specified threshhold. The failed scenarios would be printed on the console but no files would be generated. Following message would be displayed:

  More than x% of the scenarios failed. Not re-executing the failed scenarios.

updateResultFiles

Once the failed scenarios have been re-executed, call the following function to merge the re-execution results to the original execution results. It also deletes the _failedScenarios.feature and '_failedScenariosConf.js' files generated by the function identifyFailedScenarios.

retryFailedScenarios.updateResultFiles(input)

Please note that before initiating the execution, JSON results folder should be blank so that no previous execution results are used in the entire process.

TIP: gulp can be used to combine all these steps as a sequence of tasks which could then be executed against one single gulp command

Known Issues

Scenarios which contain data tables are not handled during re-execution of failure scenarios.

License

(The MIT License)

Copyright (c) 2019 Manish Kumar Singh [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Copyright (c) 2019 Manish Kumar Singh

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.