Lokalise Playwright Reporters

This is a collection of reporters to use with our Playwright test frameworks.

Key Features

• Retry reporter: Provides an easy way to retry failed tests on Jenkins or locally.
• Analytics reporter: Uploads test results and browser usage to ElasticSearch for further analysis.

Getting Started

• Add the package to your project:

npm install @lokalise/playwright-reporters

• Add the reporters to your Playwright configuration:

import { defineAnalyticsReporterConfig, defineRetryReporterConfig } from '@lokalise/playwright-reporters';

export default defineConfig({
  ...
  reporters: [
    ['retry', defineRetryReporterConfig({ maxRetryCount: 2, isPREnv: false })],
    ['analytics', defineAnalyticsReporterConfig({
        flakyTestIndex: 'flaky_tests_0001',
        failedTestIndex: 'non_passing_tests_0001',
        testRunIndex: 'test_run_0001',
        browserUsageIndex: 'browser_usage_0001',
        elasticUrl: 'http://elasticUrl',
        elasticToken: 'elasticToken',
        currentBuildId: 'buildId',
        debug: true,
        dryRun: false,
        prenvId: 'develop',
        product: 'product',
      })],
  ],
  ...
});

Steps Duration Reporter

This reporter aggregates the duration of each discrete test step and reports their total duration at the end of the test run. The reporter uses the custom Playwright Reporter implementation to collect the step durations. Follow the steps in the official Playwright documentation to add the custom reporter to your test suite.

Usage

Once the reporter is enabled, it will output a JSON file ./reporters/results/steps.json containing the duration of each step in the test suite.

Test Similarity Reporter

This reporter calculates the similarity between tests based on the test steps. The reporter does not use the custom Playwright Reporter implementation. Instead, it takes the simplified reporter from the undocumented timeline reporter.

Usage

1. Run the test suite with the timeline reporter enabled. This will generate a JSON file ./reporters/results/timeline.json
2. Run the similarity reporter with the following command: npx ts-node reporters/testSimilarity.ts
3. After analysing the results, the reporter will output a JSON file ./reporters/results/distance.json containing the similarity between tests based on the test steps.

Output

The report has the following format:

[
	{
		"title": "Title of the test being compared",
		"relatedTests": [
			["Title of the first related test", 0],
			["Title of the second related test", 50]
		]
	}
]

The relatedTests array contains the title of the related test and the distance score. The similarity percentage is calculated based on the Levenshtein distance between the test steps. The lower the distance, the higher the similarity between the tests.

Husky

By default, pre-commit hook will run npm run lint:fix. Feel free to remove that if it's undesirable or add your own pre-commit commands.

Check out husky docs for more info.

commitlint

You can use npm run commit to interactively construct correct commit messsage.

Check out commitlint docs for examples of how to customise.

Release actions

The following token needs to be set in the Github repo for the prerelease and release Github Actions to work:

• secrets.NPM_TOKEN (need this to publish on NPM)

When performing a release, make sure to follow our conventional commit approach, as described in contribution documentation.

License

This project is APACHE, VERSION 2.0 licensed, see LICENSE.md for details.