UserWay App Playwright Test

Intro

The @userway/a11y-playwright is an NPM package designed to help you perform accessibility testing on your web pages. With it you can easily run static page analysis on a webpage and get a detailed report of accessibility violations based on WCAG guidelines and ACT rules.

// playwright/e2e/my-test.spec.js

test('example test', async ({ page }) => {
  await page.goto('http://localhost:5000')
  await userwayAnalysis(page)
})

Prerequisites

Playwright version 1.16.0 or higher

Distribution

UserWay App Playwright is distributed as a zip-packaged NPM module and should be installed as an NPM dependency.

Installation

First, extract the provided userway-playwright-app.zip inside of a separate directory, like src/packages:

src
└── packages
   ├── userway-app-playwright.zip
   └── userway-app-playwright

Install userway-app-playwright with npm install:

npm install src/packages/userway-playwright-app

This adds @userway/a11y-playwright to the dependencies in package.json.

Setup

UserWay App Playwright external package and needs to be imported in playwright test.

Import @userway/a11y-playwright to your Playwright's test file and call userwayAnalysis function:

const { userwayAnalysis } = require('@userway/a11y-playwright')

test('example test', async ({ page }) => {
  await page.goto('http://localhost:5000')
  await userwayAnalysis(page)
})

Types

If you are using TypeScript, add @userway/a11y-playwright to types section in your tsconfig.json:

{
  "compilerOptions": {
    "types": ["playwright", "@userway/a11y-playwright"]
  }
}

If you are not using TypeScript, you can still have autocompletion available by adding type references to your tests:

/// <reference types="playwright" />
/// <reference types="@userway/a11y-playwright" />

// ↑ Add this at the top of your test

Usage

This example shows how to use UserWay App Playwright to run static page analysis on a webpage:

it('Evinced basic example', async ({ page }) => {
  // Visit a target page
  await page.goto(
    'we need to deploy some demo page (Evinced demo: https://demo.evinced.com/)',
  )
  // Run static analysis
  await userwayAnalysis(page)
})

By default, UserWay App Playwright scans the page for AA violations and asserts on the number of issues, but you can easily customize it's behavior.

API

userwayAnalysis(page: Page, config: AnalyzeConfig): AnalysisResult

Runs static analysis on the current page and returns a result object that contains violations. By default asserts that the number of violations is equal to zero.

Default config:

userwayAnalysis(page, {
  strict: false,
  level: 'AA',
})

To make manual assertion you can use strict: false:

userwayAnalysis(page, { strict: false }).then((res) => {
  expect(res.violations).to.have.length(0)
})

Get access to the configuration object from results:

userwayAnalysis(page, { strict: false }).then((res) => {
  console.log(res.fullReport.config)
})

Full config:

type Config = {
  excludeRules: string[] // Provide a list of rules that should be excluded
  failInapplicable: boolean // *Maybe remove*
  failIncomplete: boolean // *Maybe remove*
  strict: boolean // Set to `true` if you don't want to manually process the result without automatic assertion
  includeBestPractices: boolean // Set to `true` if you want to use best practices rules
  includeExperimental: boolean // Set to `true` if you want to use experimental rules
  includeRules: string[] // Provide a list of rules that should be included
  level: 'A' | 'AA' | 'AAA' | null // Conformance level (`null` disables all A, AA and AAA rules)
  onResult: (data: Result) => void // Hook that called when result is ready
  reportPath: string // Provide path to folder where to store artifacts ( global configuration only ) Default is "uw-a11y-reports"
  elementScreenshots: boolean // Set to `true` if you want save a screenshots of violations
  ignoreSelectors: string[] // Specify selectors for elements that should be ignored. Default is ["data-userway-app-ignore"]
  switchOff: boolean // Allows to turn off rules check without any modification of the tests
}

Examples:

// Save report and screenshots
userwayAnalysis(page, {
  elementScreenshots: true,
})

// Enable all rules
userwayAnalysis(page, {
  level: 'AAA',
  includeBestPractices: true,
  includeExperimental: true,
})

// Enable best practices and experimental rules only
userwayAnalysis(page, {
  level: null,
  includeBestPractices: true,
  includeExperimental: true,
})

// Enable rules from includeRules parameter only
userwayAnalysis(page, {
  level: null,
  includeRules: ['duplicate-id', 'color-contrast'],
})

// Disable all rules
userwayAnalysis(page, {
  level: null,
})

// Manually assert
userwayAnalysis(page).then((res) => {
  const { fullReport, violations } = res
  // Log the full report
  console.log(fullReport)
  // Assert
  expect(violations).to.have.length(0)
})

setupUserway(config: AnalyzeConfig): void

In order to avoid providing configuration for each call of userwayAnalysis function you can provide global configuration once. The best option is to set global settings in Playwright config or global setup file, but if you use setGlobalConfig please make sure that it's called only once before your tests

Note: Global configuration could be overridden by passing configuration to userwayAnalysis call in you tests

// playwright.config.ts

import { defineConfig } from '@playwright/test'
import { setupUserway } from '@userway/a11y-playwright'

setupUserway({
  reportPath: 'uw-a11y-reports',
})

export default defineConfig({}) // Playwright config definition

or add config definitions to global setup file.

// global.setup.ts

import { test as setup } from '@playwright/test'
import { setupUserway } from '@userway/a11y-playwright'

setupUserway({
  reportPath: 'uw-a11y-reports',
})

setup('do login', async ({ page }) => {
  // any specific instructions for your tests
})

Check it for more information about global setup

saveReportArtifacts(config: SaveReportConfig): void

Save violation report in manual control mode. It's handy when you need to save report conditionally, based on violations reported or your own custom logic.

saveReportArtifacts depends on report from userwayAnalysis and requires violations data to be provided.

Output format and path where to store report could be provided by configuring format and reportPath parameters.

In order to have screenshots attached to report, you need to provide screenshotsMeta to saveReportArtifacts This data can be extracted from userwayAnalysis if screenshots parameter passed in the method config.

Note: Make sure reportPath provided to saveReportArtifacts and setupUserway are the same to make screenshots be accessible in the report.

Example:

import { setupUserway } from '@userway/a11y-playwright'

setupUserway({
  reportPath: 'uw-a11y-reports',
})

userwayAnalysis(page, {
  elementScreenshots: true,
}).then((res) => {
  const { fullReport, violations, screenshotsMeta, meta } = res

  // Save JSON report
  saveReportArtifacts({ violations, meta, saveReport: 'json' })

  // Save CSV report with custom report path
  saveReportArtifacts({
    violations,
    saveReport: 'csv',
    reportPath: 'uw-a11y-reports',
  })

  // Save HTML report with screenshots of violated elements
  saveReportArtifacts({ violations, saveReport: 'html', screenshotsMeta })
})

Config

level

Specify the conformance level. Possible values: A, AA, AAA. Default is AA.

ignoreSelectors

List of selectors for elements to ignore. Accepts an array of strings. Default selector is "data-userway-app-ignore"

Note: use "ignoreSelectors" only to exclude specific elements, avoid using it in root element like body, head, html ( for such elements you can manually exclude related rules ) To see list of all rules related to html or body elements refer to "Rules" section.

ignoreUrls

Use this parameter if you need to skip a tests that open specific URLs. Accepts an array of valid RegExp. Default is undefined

Examples:

userwayAnalysis(page, {
  ignoreUrls: [/localhost:3000/, /example.com/],
  // Ignore a specific domain
})

userwayAnalysis(page, {
  ignoreUrls: [/home/, /http:/],
  // Ignore any url that contains specified string
})

userwayAnalysis(page, {
  ignoreUrls: [/home/, /settings\/privacy/],
  // Ignore a specific path
})

userwayAnalysis(page, {
  ignoreUrls: [/home\/*/, /settings\/*/],
  // Ignore a specific path and any of its subdirectories or path segments
})

reportPath

Specify custom path where to store accessibility reports and artifacts. Could be specified only in global configuration. Default is uw-a11y-reports

screenshots

Specify whether screenshots should be saved.

Note: Enabling the screenshots feature may have an impact on performance.

includeRules

List of rules to include. Accepts an array of rule IDs. Default is [].

excludeRules

List of rules to exclude. Accepts an array of rule IDs. Default is [].

Note, that this parameter has higher priority then includeRules. It means that same rule provided to includeRules and excludeRules parameters will be ignored.

includeBestPractices

Specify whether to include best practices rules. Default is false.

includeExperimental

Specify whether to include experimental rules. Default is false.

strict

Specify whether to make assertion on the number of accessibility violations. Default is false

Global configuration

In order to provide global configuration we have two options:

1. Define global config in Playwright configuration or setup file by setGlobalConfig method call.
2. Set global configuration in your Playwright test file.

Global configuration provided to special configuration command

setupUserway({ reportPath: 'uw-a11y-reports' })

Configuration override

Global configuration can be overridden, the rule of thumb is that specific method override more general definition

For example:

// global.setup.ts
import { setupUserway } from '@userway/a11y-playwright'

setupUserway({
  level: 'AA',
  ignoreUrls: [/localhost:3000/],
})

// playwright/e2e/my-test.spec.js
userwayAnalysis(page, {
  level: 'A',
  ignoreUrls: [/home/],
})

// ends up with such configuration
// {
//    level: 'A',
//    ignoreUrls: [/home/, /localhost:3000/]
// }

Note that types like arrays and objects won't be overridden, they will be merged with previous defined values

Rules

These kind of rules are reviewing html or body element, and if you want to ignore these elements - exclude the rules that check them.

• aria-hidden-body
• bypass
• bypass-heading
• bypass-landmark
• bypass-move-focus
• css-orientation-lock
• document-title
• html-has-lang
• html-lang-valid
• html-xml-lang-mismatch
• meta-viewport
• meta-viewport-large
• page-has-heading-one
• valid-lang

Reports

Document boundary selector

This selector >>> is used to represent the relation between a document and its subdocument like document - iframe, document - Shadow DOM, iframe - iframe, etc. In the report it means that violated element located inside a subdocument.

OutputType

The outputType field in violation reports can contain group as a value, and it means that violated elements grouped by issue root cause. It is very handy for violations with duplicate values ( like: ids, alt attributes, landmarks )

Examples for duplicate-id rule violation:

<div>
  <span id="one">goes to the first group</span>
  <p id="two">goes to the second group</p>
  <span id="two">also goes to the second group</span>
  <p id="one">goes to the first group</p>
</div>

In such case extra field issuesGroup added to report and contains grouped elements by ID attribute which was duplicated.

{
    issuesGroup: {
        one: [issue('span#one'), issue('p#one')],
        two: [issue('p#two'), issue('span#two')],
    }
}

Report path customization

There is a parameter reportPath from the config that can be used to customize the output path for accessibility reporting.

Also besides the parameter you can define environment variable USERWAY_CA_REPORT_PATH=your_custom_path before playwright tests execution:

USERWAY_CA_REPORT_PATH=custom-path playwright test ./dev/playwright.spec.ts

The path could be absolute or relative. In the case of the relative path, the root folder is at the same level as package.json from where playwright command is executed.

In the following example folder uw-a11y-reports will be created in the same level where playwright command is executed ( most probably with the package.json file ) The reports folder will contain HTML file with output and another folder with related screenshots ( because elementScreenshots: true passed to static analysis function )

To provide easier access to specific test results reports in the folder grouped by test's title.

Report structure

Reports are saved to a folder specified by reportPath or the default one uw-a11y-reports folder from the project root. The reports folder has the following structure:

reports - folder with accessibility reports that contain violation information and references to other artifacts. pages - folder with original HTML documentation of web pages that have been tested screenshots - folder with screenshots of found accessibility violations ( only with enabled screenshots feature )

// global.setup.ts
import { setupUserway } from '@userway/a11y-playwright'

setupUserway({
  level: 'AA',
  reportPath: 'uw-a11y-reports',
  ignoreUrls: [/localhost:3000/],
})

// playwright/e2e/my-test.spec.js
await userwayAnalysis(page, {
  elementScreenshots: true,
})

Error messages and recommendations

Each violation in the report contains two fields errorMessage and recommendation that refer to WCAG standard and provided for better understanding of the violation's root cause with the guidelines of how to fix it.

Violation variations

Some violations might have a few root causes. For example, the rule image-alt can be violated by missed alt attribute or by having it unfilled with an empty string. The field variant in the violated element selector points to related information described inerrorMessage and recommendation that helps to identify the root cause and appropriate fixes more granularly.

Global Switch

Global Switch allows to switch off or switch on accessibility analysis. It could be needed, for example, while debugging tests in your local environment with or without static analysis tool.

There are two options to switch functionality off:

1. Provide switchOff: true config parameter to global configuration by setupUserway method call.

// playwright.config.ts

import { defineConfig } from '@playwright/test'
import { setupUserway } from '@userway/a11y-playwright'

setupUserway({ switchOff: true })

export default defineConfig({}) // Playwright config definition

Note: switchOff parameter can be also specified for userwayAnalysis method.

2. Define environment variable USERWAY_CA_SWITCH_OFF=true before playwright tests execution:

USERWAY_CA_SWITCH_OFF=false playwright test ./dev/playwright.spec.ts