cypress-multithreaded-runner

A lightweight method to run Cypress spec files in multiple threads, with built-in support for Allure report generation.

When used out of the box, Cypress will only run your spec files in a single thread. It also won't create any kind of easy-to-read report once the tests have completed. By making use of this module, you'll be able to adapt your Cypress project to run much faster than before, while also generating a comprehensive Allure report for every run.

NOTE: Currently, generating the Allure report is needed in order for this module to function correctly, but this may change later.

Prerequisites

• node 18 or above
• Java
• A Cypress project (tested on versions 12.10 and 13.7)
• @cypress/grep (optional: only needed if you want to use Cypress grep features)

Setup

In its current form, you can import the runner into a simple node application, initialising it like so:

const runner = require("cypress-multithreaded-runner");

const cypressConfigOverride = {
  reportDir: "put-the-reports-here",
};

runner({
  phaseDefaults: {
    cypressConfig: {
      filepath: "src/cypress/configurations/my-generic-cypress.config.js",
      object: cypressConfigOverride,
    },
    grep: "my-test",
    grepTags: "my-tag",
  },
  phases: [
    {
      specsDir: "src/cypress/tests/phase1",
    },
    {
      specsDir: "src/cypress/tests/phase2",
    },
  ],
  allureReportDir: "i-would-rather-save-the-allure-report-here",
  reportDir: cypressConfigOverride.reportDir,
  waitForFileExist: {
    filepath: "a-file-that-I-expect-to-exist-before-subsequent-threads-run.txt",
    timeout: 60,
    deleteAfterCompletion: true,
  },
});

By default, every spec file within each phase's specsDir will spawn a unique instance of Cypress. Once everything's been tested, the Allure report will be generated.

A benchmark file will also be generated. This will dictate the optimal order in which the threads should spawn in future. Some identifiers from your config are used to determine which order should be used. That way, you can run your Cypress tests with different arguments (for example, different grep arguments) and the results will not interfere with other benchmarks.

Should tests in any phase fail, all threads from subsequent phases will stop immediately. If you don't wish this to happen, you'll want to just use one phase.

Generating the Allure report

To generate Allure reports, you will need to add the following import to your support file:

import "cypress-multithreaded-runner/allure";

In addition, add the following import to your config file:

const allureWriter = require("cypress-multithreaded-runner/allure/writer");

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      allureWriter(on, config);
      return config;
    },
  },
});

The imports above are simple wrappers for @mmisty/cypress-allure-adapter, so refer to that module's readme should you wish to customise the plugin's behaviour.

If either of the imports are missing, you may find that an empty Allure report is generated after running tests.

Optional: Grep

If you want to make use of @cypress/grep, add it as a dependency to your project and follow the official readme to see how to import it. Refer to the options table to see what features are supported by cypress-multithreaded-runner out of the box.

Using grep in conjunction with onlyRunSpecFilesIncludingAnyText and onlyRunSpecFilesIncludingAllText

Grep can be a little slow if you have a lot of spec files, as Cypress will still try to run each of them before evaluating whether they satisfy the grep rules.

To save time, you may want to use utilise onlyRunSpecFilesIncludingAnyText and/or onlyRunSpecFilesIncludingAllText in addition to grep. These properties will ensure that spec files will only be processed if the given string(s) are found within them. Unlike grep, the filtering will occur before Cypress is launched, which in practice will mean you can expect a significant speed boost in many scenarios. However, as it's only checking for the presence of strings, you shouldn't pass through advanced grep rules to onlyRunSpecFilesIncludingAnyText and/or onlyRunSpecFilesIncludingAllText.

While it may be a common use case, you don't need to use grep in order for onlyRunSpecFilesIncludingAnyText or onlyRunSpecFilesIncludingAllText to function. The values you assign to these properties do not need to be the same as the ones you give to grep or grepTags, as it works completely independently.

For basic grep rules, the following example may represent a common use case:

const runner = require("cypress-multithreaded-runner");

const cypressConfigOverride = {
  reportDir: "put-the-reports-here",
};

runner({
  cypressConfig: {
    filepath: "src/cypress/configurations/my-generic-cypress.config.js",
    object: cypressConfigOverride,
  },
  reportDir: cypressConfigOverride.reportDir,
  specsDir: "src/cypress/tests",
  grep: "my-test",
  grepTags: "my-tag",
  onlyRunSpecFilesIncludingAllText: ["my-test", "my-tag"],
  ignoreCliOverrides: ["grep", "grepTags"],
});

Optional: Prevent the parent process from ending when tests fail

By default, this module will end the parent process (with exit code 1) should the tests complete with one or more failures. Bypassing this will therefore allow your app to continue running every time the tests complete. You can set endProcessIfTestsFail to false to achieve this.

Optional: Generate JUnit reports & upload to BrowserStack Test Observability

An additional JUnit report can be generated alongside Allure. This report can also be uploaded to BrowserStack Test Observability, if you have an account with them:

const runner = require("cypress-multithreaded-runner");

const cypressConfigOverride = {
  reportDir: "put-the-reports-here",
};

runner({
  cypressConfig: {
    filepath: "src/cypress/configurations/my-generic-cypress.config.js",
    object: cypressConfigOverride,
  },
  reportDir: cypressConfigOverride.reportDir,
  specsDir: "src/cypress/tests",
  jUnitReport: {
    enabled: true,
    browserStackTestObservabilityUpload: {
      username: "my-browserstack-username",
      accessKey: "my-browserstack-key",
      parameters: {
        projectName: "my-project-name",
        buildName: "my-build-name",
      },
    },
  },
});

Get the exit code

Unless an error unrelated to one of your tests occurs, you can get the aforementioned exit code by running this module in async mode. To achieve this, import and use the module like so:

const asyncRunner = require("cypress-multithreaded-runner/async"); // different import

const cypressConfigOverride = {
  reportDir: "put-the-reports-here",
};

(async () => {
  const exitCode = await runner({
    cypressConfig: {
      filepath: "src/cypress/configurations/my-generic-cypress.config.js",
      object: cypressConfigOverride,
    },
    reportDir: cypressConfigOverride.reportDir,
    specsDir: "src/cypress/tests",
    endProcessIfTestsFail: false, // if this isn't set to false, your app will stop running if any tests fail!
  });

  console.log("exitCode: ", exitCode);
})();

The exit code will be 0 if all tests pass and 1 if any test fails.

General guidance

Pay attention to the performance of each thread when adding a new test. It is a good idea to try making the accumulation of tests in each thread take roughly the same amount of time to finish running, as any thread that takes significantly longer than the others will be a bottleneck! Therefore, try to add any new tests to threads which aren't oversaturated. You can have a look at how each thread is performing by viewing the "Thread Performance Summary" & other logs, all of which will be added to the bottom of the Allure report as well as in separate text files.

Diminishing returns will be observable the more threads you add, as most computers have a relatively low number of threads that can be actively used at any given time. When too many threads are added you may notice a large increase in the frequency of failing tests, and you may also actually notice that other threads are just running slower in general. It's all a bit of a balancing act if you want to get it all just right.

None of the Cypress threads can "talk" to each other, so your project may need a bit of extra configuration if this is needed. For example, let's say your first thread logs in to a website and you'd like this session to be maintained across your other threads. You could have the first Cypress thread log in and then save a file containing all of the cookies that the other threads need. You can make use of the waitForFileExist option to help make such functionality possible for your project. cypress-wordpress-session is an example of a Cypress addon that can be used to create & load a cookies file for a persistent session across multiple threads.

Overriding options in the command line

The module makes use of argv to allow options to be overriden via the command line. Arguments passed in via the command line will take precedence over any options set via your node application. For example, let's say you want to override the reportDir option. You can run your node application with a standard kebab-case argument (eg. --report-dir="my-report-dir") and the reportDir will be overriden.

Options

| Name | Type | Default value | Description | | --------------------------------- | ---------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | reportDir | string | null | The location to save the full report to. This currently needs to be the same directory that's passed through to the Cypress instances. | | clean | boolean | false | Delete all files in the reportDir before starting the threads | | allureReportDir | string | <reportDir>/allure-report | The location to save just the Allure report | | allureHistoryDir | string | <reportDir>/allure-report/history | The location to retrieve & overwrite just the Allure history. | | overwriteAllureHistory | boolean | true | Copy the Allure history to the allureHistoryDir after the tests have run. Useful only if the allureHistoryDir is different to default location. | | allureReportHeading | string | null | A custom heading to add to the top of the Allure report | | generateAllure | boolean | true | Generate the Allure report. Overrides the openAllure, combineAllure & hostAllure options if set to false. NOTE: It's not currently possible to completely disable this feature. For now, setting this to false will generate only what's required for this module to function correctly. | | generate | boolean | true | CLI alias for generateAllure | | openAllure | boolean | false | Open the Allure report after it's generated.If combineAllure is also passed in, it'll open the combined Allure report instead.If hostAllure is also passed in, openAllure will be ignored. | | open | boolean | false | CLI alias for openAllure | | combineAllure | boolean | true | Combine the Allure report into a single file (complete.html). | | combine | boolean | true | CLI alias for combineAllure | | hostAllure | boolean | false | Spin up a localhost for the Allure report after it's generated | | host | boolean | false | CLI alias for hostAllure | | ignoreCliOverrides | arrayOf(string) | null | A list of keys of properties that you don't want the CLI to override when you run an instance of cypress-multithreaded-runner. This will enable your node app to do a custom override of that uses a combination of the CLI and itself. See here for an example | | specFiles | arrayOf(string) | null | An array of one or more spec files to filter. Only spec files present in the array will be tested. This will apply to every phase, so you may find some phases are skipped entirely. The files must all exist within the given specsDir directory. | | specs | arrayOf(string) | null | CLI alias for specFiles | | runFailedSpecFilesFromReportURL | string | null | Provide a URL (remote or local) of an Allure report generated by Cypress Multithreaded Runner (v1.15 and above) to only run the spec files that failed in that particular run. | | runFailedSpecsFromReportURL | string | null | CLI alias for runFailedSpecFilesFromReportURL | | runFailedFromReportURL | string | null | CLI alias for runFailedSpecFilesFromReportURL | | runFailsFromReportURL | string | null | CLI alias for runFailedSpecFilesFromReportURL | | logMode | oneOf([1,2,3,4]) | 1 | The method by which you want each Cypress thread to print logs to the console.1: Only print logs from one thread at a time, in order of when each thread began. Only when the first thread completes will the logs from the second thread be printed, then the third and so on.2: Only print logs from one thread at a time, but allow non-chronology. Should any other thread complete before the current one does, these logs will be "queued" and then print when the current one completes. For example, let's say thread 3 completes before thread 1 completes, with thread 2 completing at the end. In this scenario, you can expect to see all of the output for thread 1, then all of thread 3, then thread 2.3: Print any log from any thread as soon as it manifests. This will likely mean that you see a mix of logs from each thread, and can be difficult to understand the continuity in the logs the more threads you have.4: Identical to mode 3, except that when all threads complete, print all logs again but separate out all of the threads. | | waitForFileExist | object | null | Wait for a specific file to exist (and larger than 0 bytes in size) before subsequent threads begin. For specific options, see the table below. | | maxThreadRestarts | number | 5 | Should an instance of Cypress crash, it may be restarted up to this many times until all threads complete successfully. Note that any spec file that fails in a beforeEach hook will be considered a crash. This behaviour may be amended in a future version of this module. | | maxConcurrentThreads | number | Half the number of available threads | The maximum number of threads that'll run at any one time. By default, this will be the number of threads the client's CPU is reporting divided by two. It's recommended not to use more threads than a CPU can provide, otherwise performance is likely to be negatively affected. If not enough threads are available, cypress-multithreaded-runner will wait for any of the currently running Cypress instances to complete before starting another one. This process will repeat until all tests are completed. | | notify | boolean | true | Fire a platform native notification when tests have completed | | threadDelay | number | 30 | The amount seconds to wait before starting the next thread, unless the current Cypress instance has already started running. If waitForFileExist has been set, the 2nd thread will continue waiting until the given file exists. For more info, see the table below. | | threadMode | oneOf([1,2]) | 1 | The method by which you want each Cypress thread to be scheduled logs to the console.1: Every spec file runs in its own Cypress instance. This is recommended for most projects. If there's a lot of variance in how fast each spec file takes to run, then this is the best solution for you.2: A thread will be created for every top-level directory within your specsDir. For example, 8 top-level folders will be 8 threads. There's a small time cost in spinning up a new Cypress instance, so if you want to spend time manually shuffling the spec files around to balance each thread, you may find a performance gain using this instead of mode 1. NOTE: If you make use of the specFiles option, threadMode will always be set to mode 1. | | onlyPhaseNo | number | null | Set this to the value of any of the phases to run just the threads within that phase. Phases count from a value of 1, not 0. | | startingPhaseNo | number | null | Set this to the value of any of the phases to run just the threads within and after that phase. Phases count from a value of 1, not 0. | | endingPhaseNo | number | null | Set this to the value of any of the phases to run just the threads before and within that phase. Phases count from a value of 1, not 0. | | threadInactivityTimeout | number | 600 | The maximum amount of seconds to wait for a thread to respond before it's considered a crash. Every time any thread logs something, the timeout will be reset. Set to 0 to have no timeout at all. | | threadTimeLimit | number | 1800 | The maximum amount of seconds to wait for a thread to complete. This is the total allocated time per thread. It won't reset should the thread restart due to errors. Set to 0 to have no time limit at all. | | orderThreadsByBenchmark | boolean | true | Honour the order of the threads as set in the thread benchmark file. If no such file exists, it'll fallback to alphabetical order. Any threads that don't exist in the benchmark file will run at the start of the phase. | | saveThreadBenchmark | boolean | true | Update the thread benchmark file with a new order for the threads, with the slowest thread at the start and the fastest thread at the end. | | threadBenchmarkFilepath | string | cmr-benchmarks.json | The file where the thread benchmark will be saved. | | benchmarkDescription | string | null | An optional string to add to the benchmark such that its identifier is easily understandable. The value of this string will be the sole identifier for the benchmark. For example, if you run your suite of tests two times and the only argument that's different is the benchmarkDescription, the results from the first run won't overwrite the second, nor will the results from the first run be used to order the threads in the second run. If this field is left empty, the benchmark will instead be automatically identified by means of encoding a string comprised of various other attributes, such as some of the phase properties. | | jUnitReport | object | null | Generate a JUnit report for all threads. Separate XML files will be generated for each thread and then combined into one. This can then be uploaded to BrowserStack Test Observability. See table below | | phaseDefaults | object | null | Default properties you wish to set for every object in the phases array. For more info, see table below. | | phases | arrayOf(object) | null | Phases of Cypress test threads. Any phase can have any number of threads. Every object will override equivalent property keys set in phaseDefaults.Should tests in any phase fail, all threads from subsequent phases will stop immediately. For example, you may want to run high priority tests first, then medium priority, then low priority. If the high priority tests fail, the medium & low priority tests will stop running. For more info, see table below. | | repeat | number | 1 | The number of times all phases should repeat. Make use of this to stress test your system and identify any unstable tests. | | endProcessIfTestsFail | boolean | true | When set to true, the parent process will stop running (with exit code 1) if any test fails. | | maxConcurrentThreadsExperiment | object | null | An experimental feature in name and function! This can be used to determine the optimum number of threads that can be used to run Cypress tests on your machine. It does this by running all of your tests with a different value set for maxConcurrentThreads and then comparing the time taken for each setting. For specific options, see table below.NOTE: This feature will not work if the module is run in the async mode. |

phases

| Name | Type | Default value | Description | | ---------------------------------- | ---------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | cypressConfig | object | null | The config that'll be passed through to every Cypress instance. For specific options, see the table below. | | specsDir | string | null | The top-level directory containing all Cypress spec files are | | browser | string | null | The web browser you want the spec files to use | | grep | string | null | grep arg to be passed through as an environment variable to each Cypress instance. See here for more information. | | grepTags | string | null | grepTags arg to be passed through as an environment variable to each Cypress instance. See here for more information. | | grepUntagged | boolean | false | grepUntagged arg to be passed through as an environment variable to each Cypress instance. See [here](https://www.npm