TestRail Reporter for Cypress

Publishes Cypress runs on TestRail.

Bug fixes

• Users migrating to Cypress 10 can now export screenshots and videos

New Features

• Possibility to upload videos for failed test cases - optional (allowOnFailureVideoUpload: true)
• Possibility to upload downloads folder for all test cases - optional (allowExportDownloads: true)
• Possibility to aggreagate multiple sections of a same test suite to one test run (see groupId option)
• Possibility to precise the runId of a manually created test run on Testrail(see runId option)

Core features:

• No dependency from deasync
• Test results are aggregated under the same test run if you are executing more spec(test) files and they are belongs to the same suite
• Results are reported immediately after single test execution (real-time reporting) except the video recording which is reported at the end
• Possibility to upload screenshots for failed test cases - optional (allowOnFailureScreenshotUpload: true)
• Multi suite project support (set suiteId=1 in cypress.json or set it as a part of runtime environment variables as testRailSuiteId=1)
• Reporting retest status of a test cases - handy in terms of marking tests as flaky (test is reported with retest status for the first try and after second try it passes) Note: cypress retry logic must be enabled for this feature.

Install

$ npm install @mkonate/cypress-testrail-reporter --save-dev

Usage

Add reporter to your cypress.json:

...
"reporter": "cypress-testrail-reporter",
"reporterOptions": {
  "host": "https://yourdomain.testrail.com",
  "username": "username",
  "password": "password",
  "projectId": 1,
  "suiteId": 1,
}

Your Cypress tests should include the ID of your TestRail test case. Make sure your test case IDs are distinct from your test titles:

// Good:
it("C123 C124 Can authenticate a valid user", ...
it("Can authenticate a valid user C321", ...

// Bad:
it("C123Can authenticate a valid user", ...
it("Can authenticate a valid userC123", ...

Reporter Options

host: string host of your TestRail instance (e.g. for a hosted instance https://instance.testrail.com).

username: string email of the user under which the test run will be created. When you set CYPRESS_TESTRAIL_REPORTER_USERNAME in environment variables, this option would be overwritten with it.

password: string password or the API key for the aforementioned user. When you set CYPRESS_TESTRAIL_REPORTER_PASSWORD in runtime environment variables, this option would be overwritten with it.

projectId: number project with which the tests are associated.

suiteId: number suite with which the tests are associated. Optional under cypress.json file in case that you define suiteId under gitlab-ci.yml file or set this value in runtime environment varables.

runName: string (optional) name of the Testrail run. When you set CYPRESS_TESTRAIL_REPORTER_RUNNAME in runtime environment variables, this option would be overwritten with it.

disableDescription: bool (optional: default is false) possibility to disable description for test run in case that someone don’t have cypress dashboard feature (disableDescription: true)

allowOnFailureScreenshotUpload: bool (optional: default is false) will upload failed screenshot to corresponding test result comment for easier debugging of failure. (Required: screenshotOnRunFailure option must be set to true in cypress.json )

allowOnFailureVideoUpload: bool (optional: default is false) will upload a video of the test to the corresponding test result comment. (Required: video option must be set to true in cypress.json )

allowExportDownloads : bool (optional: default is false) will upload the cypress downloads folder to the test run. (Note : This setting is useful if you are downloading files in your automated tests. Be advised that you have the resposability to cleanup your cypress downloads folder)

includeAllInTestRun: bool (optional: default is true) will return all test cases in test run. set to false to return test runs based on filter or section/group.

groupId: string (optional: needs "includeAllInTestRun": false ) A comma separated list of IDs of the sections/groups. When you set CYPRESS_TESTRAIL_REPORTER_GROUPID in runtime environment variables, this option would be overwritten with it.

filter: string (optional: needs "includeAllInTestRun": false) Only return cases with matching filter string in the case title

runId: number will aggregate the test cases to the test run ID entered. When you set CYPRESS_TESTRAIL_REPORTER_RUNID in runtime environment variables, this option would be overwritten with it.

Multiple suite

This reporter can handle multiple suite project in TestRail. In order to use it, don't define suiteId under cypress.json file and instead you should pass testRailSuiteId variable when you define all other CLI agruments for cypress execution(through command line). If you are using CI integration solution (e.g. GitLab) testRailSuiteId can be set before every pipeline job or predefined for each spec (test) file for which suiteId belongs to.

gitlab-ci.yml file (Here you can pass suiteId as a variable):

e2e_test1:
  script:
    - e2e-setup.sh
  variables:
    CYPRESS_SPEC: "cypress/integration/dashboard/*"
    TESTRAIL_SUITEID: 1

e2e_test2:
  script:
    - e2e-setup.sh
  variables:
    CYPRESS_SPEC: "cypress/integration/login/*"
    TESTRAIL_SUITEID: 2

and use it later during cypress run:

e2e-setup.sh file

CYPRESS_OPTIONS="baseUrl=${url},trashAssetsBeforeRuns=false,video=${video},screenshotOnRunFailure=${screenshotOnRunFailure}"
CYPRESS_ENV="testRailSuiteId=${TESTRAIL_SUITEID}"

npx cypress run --headed --browser chrome --config "${CYPRESS_OPTIONS}" --env="${CYPRESS_ENV}" --spec "${CYPRESS_SPEC}"

Cucumber preprocessor

This reporter can miss spec files if they are suffixed as .feature or if you are not using the default cypress/integration folder. In order to use it with the Cucumber Preprocess, you should pass the location of your spec files cypress/tests/**/*.feature when you define all other CLI agruments for cypress execution(through command line). If you are using CI integration solution (e.g. GitLab) CYPRESS_SPEC can be set before every pipeline job

CYPRESS_SPEC="cypress/tests/**/*.feature"

npx cypress run --headed --browser chrome --spec "${CYPRESS_SPEC}"

TestRail Settings

To increase security, the TestRail team suggests using an API key instead of a password. You can see how to generate an API key here.

If you maintain your own TestRail instance on your own server, it is recommended to enable HTTPS for your TestRail installation.

For TestRail hosted accounts maintained by Gurock, all accounts will automatically use HTTPS.

You can read the whole TestRail documentation here.

Author

Matt Charlton- github

Core contributors

• Milutin Savovic
• Anes Topcic
• FFdhorkin

License

This project is licensed under the MIT license.

Acknowledgments

• Pierre Awaragi, owner of the mocha-testrail-reporter repository that was forked.
• Valerie Thoma and Aileen Santos for proofreading the README.md file and making it more understandable.
• Milutin Savovic, creater of the the original cypress-testrail-reporter