karma-accessibility-checker

v3.1.78

Published

7 days ago

A Karma plugin for Accessibility Testing.

Downloads

6,563

0High
0Medium
0Low

thbrunet

eatools

shunguoy

ci integration continuous karma-ibma karma karma-plugin a11y accessibility IBMa karma-framework karma-reporter karma-preprocessor

karma-accessibility-checker

Overview

karma-accessibility-checker is a Karma plugin that allows you to perform the following:

integrate accessibility testing within a continuous integration pipeline, such as Travis CI.
scan HTML nodes/widgets, URLs, local files, HTML documents, and allows you to scan HTML content in the form of a string
aside from just performing accessibility scanning, it provides a framework to validate accessibility scan results against baseline files and/or simply failing the test cases based on the levels of violations found during the scan

The Karma plugin is a component of the IBM Equal Access Toolkit. The Toolkit provides the tools and guidance to create experiences that are delightful for people of all abilities. The guidance is organized by phase, such as Plan, Design, Develop, and Verify, and explains how to integrate the automated testing tools into the Verify phase. The Toolkit is a major part of the accessibility information and applications at ibm.com/able.

karma-accessibility-checker

Usage

The tools that have been deployed to NPM so it can be easily downloaded and installed:

karma-accessibility-checker: automated accessibility testing for the Karma environment

Quick start

Grab a boilerplate

Getting started

Setup and initialize - Follow the instructions in the Prerequisites and Install sections.
Configure Karma and the karma-accessibility-checker plugin - Follow the Configuration instructions.
Learn how to use the karma-accessibility-checker APIs to perform accessibility scans - Refer to the Usage and API documentation.

Prerequisites

Install Node.js and NPM

Note: Use the latest available Node.js version.

Basic understanding of testing with Karma testrunner

Install

Note: For Windows, you may have to install and use cygwin CLI with curl package in order to run curl commands on windows.

Install npm

Install the karma-accessibility-checker plugin from the npm repository:

$ npm install --save-dev karma-accessibility-checker

Configuration

Karma needs to be configured before you load the karma-accessibility-checker plugin - and then you need to configure the karma-accessibility-checker plugin before using it.

Configuring Karma

Configuring `plugins`

If you are using the plugins array in the Karma configuration file (karma.conf.js), you will need to add this to the plugins array to load the karma-accessibility-checker plugin.

// karma.conf.js
module.exports = function (config) {
    config.set({
        // Plugins for karma to load
        plugins: [require("karma-accessibility-checker")],

        // ...
    });
};

Configuring `framework` and `reporters`

Add aChecker to the framework, reporters array in the Karma configuration file (karma.conf.js). This will load the karma-accessibility-checker plugin.

// karma.conf.js
module.exports = function (config) {
    config.set({
        // Frameworks to use to run the tests that we define
        frameworks: ["jasmine", "aChecker"],

        // Test results reporter to use
        reporters: ["progress", "aChecker"],

        // ...
    });
};

Configuring `preprocessor`

Use aChecker in preprocessor to process the baseline files and load them into memory. This saved copy of the files can then be used for accessibility scan results comparison.

// karma.conf.js
module.exports = function (config) {
    config.set({
        // ...

        // Preprocess matching files before serving them to the browser
        // Note: Preprocessors may be transforming the files and file types that are available at run time.
        preprocessors: {
            "test/baseline/**/*.json": ["aChecker"],
        },

        // List of files / patterns to load in the browser
        // Note: Order matters, the order they are listed here is how they are loaded.
        files: ["test/baseline/**/*.json", "test/**/*.js"],

        // ...
    });
};

Configuring the plugin

Configuring the karma-accessibility-checker plugin involves constructing a .achecker.yml file in the project root. This file will contain all of the configuration options for karma-accessibility-checker. This is the structure of the .achecker.yml file:

# optional - Specify the rule archive
# i.e. For march rule archive use ruleArchive: 2017MayDeploy
# Default: latest
# If "latest", will use the latest rule release
# If "versioned" (supported in 3.1.61+), will use the latest rule release at
# the time this version of the tool was released 
# Refer to README.md FAQ section below to get the rule archive ID.
ruleArchive: latest

# optional - Specify one or many policies to scan.
# i.e. For one policy use policies: IBM_Accessibility_2017_02
# i.e. Multiple policies: IBM_Accessibility,WCAG_2_1 or refer to below as a list
# Default: null (all policies)
# Refer to README.md FAQ section below to get the policy ID.
policies:
    - IBM_Accessibility

# optional - Specify one or many violation levels on which to fail the test
#            i.e. If specified violation then the testcase will only fail if
#                 a violation is found during the scan.
# i.e. failLevels: violation
# i.e. failLevels: violation,potential violation or refer to below as a list
# Default: violation, potentialviolation
failLevels:
    - violation
    - potentialviolation

# optional - Specify one or many violation levels which should be reported
#            i.e. If specified violation then in the report it would only contain
#                 results which are level of violation.
# i.e. reportLevels: violation
# i.e. reportLevels: violation,potentialviolation or refer to below as a list
# Default: violation, potentialviolation, recommendation, potentialrecommendation, manual
reportLevels:
    - violation
    - potentialviolation
    - recommendation
    - potentialrecommendation
    - manual

# Optional - In what format types the results should be output (json, html)
# Default: json
outputFormat:
    - json

# Optional - Should the timestamp be included in the filename of the reports?
# Default: true
outputFilenameTimestamp: true

# Optional - Specify labels that you would like associated with your scan
#
# i.e.
#   label: Firefox,master,V12,Linux
#   label:
#       - Firefox
#       - master
#       - V12
#       - Linux
# Default: N/A
label:
    - master

# optional - Where the scan results should be saved.
# Default: results
outputFolder: results

# optional - Where the tool can read/write cached files (ace-node.js / archive.json)
# Default: `${os.tmpdir()}/accessibility-checker/`
cacheFolder: /tmp/accessibility-checker

API-based usage

karma-accessibility-checker is solely an API-based Karma plugin, therefore APIs should be used within test cases on a browser launched by Karma.

To perform an accessibility scan within your test cases and verify the scan results:

// Perform the accessibility scan using the aChecker.getCompliance API
aChecker.getCompliance(testDataFileContent, testFile, function (results) {
    // Call the aChecker.assertCompliance API which is used to compare the results with the baseline object if we can find one that
    // matches the same label which was provided.
    var returnCode = aChecker.assertCompliance(results);

    // In the case that the violationData is not defined then trigger an error right away.
    expect(returnCode).toBe(
        0,
        "Scanning " + testFile + " failed." + JSON.stringify(results)
    );

    // Mark the testcases as done, when using jasmine as the test framework.
    done();
});

Refer to Examples for sample usage scenarios.

API

aChecker.getCompliance(`content`, `label`, `callback`)

Execute accessibility scan on provided content. The content can be in these forms:

HTML (String)
Single node/widget (HTMLElement)
Local file path (String)
URL (String)
Document node (HTMLDocument)

Use a callback mechanism (callback) to extract the results and perform assertion using accessibility-checker APIs.

content - (String | HTMLElement | HTMLDocument) content to be scanned for accessibility violations.
label - (String) unique label to identify this accessibility scan from others. Using "/" in the label allows for directory hierarchy when results are saved.
callback - (Function) callback that is invoked (indicating success). The callback will be invoked with the results as a parameter to the callback function provided. This is the outline of the results object which will be passed to the callback function:

{
    report: {
        scanID: "18504e0c-fcaa-4a78-a07c-4f96e433f3e7",
        toolID: "accessibility-checker-v3.0.0",
        // Label passed to getCompliance
        label: "MyTestLabel",
        // Number of rules executed
        numExecuted: 137,
        nls: {
            // Mapping of result.ruleId, result.reasonId to get a tokenized string for the result. Message args are result.messageArgs
            "WCAG20_Html_HasLang": {
                "Pass_0": "Page language detected as {0}"
            },
            // ...
        },
        summary: {
            URL: "https://www.ibm.com",
            counts: {
                violation: 1,
                potentialviolation: 0,
                recommendation: 0,
                potentialrecommendation: 0,
                manual: 0,
                pass: 136,
                ignored: 0
            },
            scanTime: 29,
            ruleArchive: "September 2019 Deployment (2019SeptDeploy)",
            policies: [
                "IBM_Accessibility"
            ],
            reportLevels: [
                "violation",
                "potentialviolation",
                "recommendation",
                "potentialrecommendation",
                "manual"
            ],
            startScan: 1470103006149
        },
        results: [
            {
                // Which rule triggered?
                "ruleId": "WCAG20_Html_HasLang",
                // In what way did the rule trigger?
                "reasonId": "Pass_0",
                "value": [
                    // Is this rule based on a VIOLATION, RECOMMENDATION or INFORMATION
                    "VIOLATION",
                    // PASS, FAIL, POTENTIAL, or MANUAL
                    "PASS"
                ],
                "path": {
                    // xpath
                    "dom": "/html[1]",
                    // path of ARIA roles
                    "aria": "/document[1]"
                },
                "ruleTime": 0,
                // Generated message
                "message": "Page language detected as en",
                // Arguments to the message
                "messageArgs": [
                    "en"
                ],
                "apiArgs": [],
                // Bounding box of the element
                "bounds": {
                    "left": 0,
                    "top": 0,
                    "height": 143,
                    "width": 800
                },
                // HTML snippet of the element
                "snippet": "<html lang=\"en\">",
                // What category is this rule?
                "category": "Accessibility",
                // Was this issue ignored due to a baseline?
                "ignored": false,
                // Summary of the value: violation, potentialviolation, recommendation, potentialrecommendation, manual, pass
                "level": "pass"
            },
            // ...
        ]
    }
}

Refer to the actualResults parameter in aChecker.assertCompliance API for more details about the properties of the results object.

aChecker.assertCompliance(`actualResults`)

Assertion will be performed one of the following ways based on the condition that is met:

If a baseline file of scan results is available in memory, it will be compared to the actualResults. If the actualResults matches the baseline, it will return 0. If not, it will return 1. In this case, assertion is only run on the XPath and ruleId.
If there is no baseline file, assertion will be made based on the provided failLevels. In this case, 2 is returned if there are failures based on failLevels.

actualResults - (Object) results for which assertion needs to be run. Properties include:
- scanID, (String) auto generated UUID used to associate a session.
- toolID, (String) tool ID for the tool that generated these results.
- label, (String) label provided to identify unique scan results. (provided through the aChecker.getCompliance API).
- URL, (String) contains the URL that was scanned using aChecker.getCompliance API. (provided through aChecker.getCompliance API).
- issueMessages, (Object) violation messages based on language.
  - messages, (Object) violation messages indexed by message ID.
  - lang, (String) locale of the violation messages.
- summary, (Object) summary of the scan that these results are for. Properties include:
  - counts, (Object) number of violations based on the violation level. Properties include:
    - violation, (Int) total number of violations.
    - potentialviolation, (Int) total number of potential violations.
    - recommendation, (Int) total number of recommendations.
    - potentialrecommendation, (Int) total number of potential recommendations.
    - manual, (Int) total number of manual checks.
  - scanTime, (Int) total number of milliseconds it took to perform the scan.
  - ruleArchive, (String) rule archive used for this particular scan result.
  - policies, (Array) policies used for the scan result.
  - reportLevels, (Array) list of violation levels to include in the report. (save to file).
  - startScan, (Int) start time of the scan in milliseconds since epoch, GMT.
- reports, (Array) list of reports in the case of multiple iframes are present on the single page. (iframe scanning not supported yet) Each array element is an object with these properties:
  - frameIdx, (Int) frame index in the page represented as an integer value.
  - frameTitle, (String) title of the frame on the page that was scanned.
  - issues, (Array) detailed list of violations. Each array element is an object with these properties:
    - severityCode, (String) severity code of the violation.
    - messageCode, (String) message code of the violation. Used to map the localized message string.
    - ruleId, (String) rule ID of the violation.
    - help, (String) help file name of the violation.
    - msgArgs, (Array) list of tokens to help provide a detailed error description of the issue.
    - bounds, (Object) provides the pixel position of the element that triggered the violation. Properties include:
      - left, (Int) left pixel position of the element.
      - top, (Int) top pixel position of the element.
      - height, (Int) height pixel position of the element.
      - width, (Int) width pixel position of the element.
    - level, (String) level of the violation.
    - xpath, (String) XPath of the element that triggered the violation.
    - snippet, (String) snippet for the element that triggered the violation.
Returns 0 if actualResults matches baseline or if no violations match the failLevels
Returns 1 if actualResults do not match the baseline
Returns 2 if there is a failure based on failLevels
Returns -1 if an exception has occurred during scanning and the results reflect that

aChecker.getDiffResults(`label`)

Retrieve the diff results for a specified scan (denoted by its label) when API aChecker.assertCompliance(...) returns 1 (when the actualResults do not match the baseline).

label - (String) label to use when getting the diff results. The label should match the one provided for aChecker.getCompliance(...).

Returns a diff object, where the left hand side (lhs) is actualResults and the right hand side (rhs) is baseline. Refer to the deep-diff documentation for the format of the diff object, and how to interpret the object.

Returns undefined if there are no differences.

aChecker.getBaseline(`label`)

Retrieve the baseline result object based on the label provided.

label - (String) label for which to get the baseline. The label should match the one provided for aChecker.getCompliance(...).

Returns object that follows the same structure as the results object outlined in the aChecker.getCompliance and aChecker.assertCompliance API.

Returns undefined if a baseline is not found for the label provided.

aChecker.diffResultsWithExpected(`actual`, `expected`, `clean`)

Compare provided actual and expected objects and get the differences between them.

actual - (Object) actual results that you want to compare. Refer to aChecker.assertCompliance for details about available properties to include.
expected - (Object) expected results to compare to. Refer to aChecker.assertCompliance for details about available properties to include.
clean - (boolean) clean the actual and expected results by converting the objects to match with a basic compliance compare of only xpath and ruleID.

Returns undefined if there are no differences.

aChecker.stringifyResults(`results`)

Retrieve the readable stringified representation of the scan results.

results - (Object) results which need to be stringified. Refer to aChecker.assertCompliance for details about available properties to include.

Returns a String representation of the scan results which can be logged to a console.