grunt-code-coverage-enforcer

Enforces code coverage thresholds for a project with lcov files. This means it works with a wide range of code coverage tools like blanketjs, istanbul, .....

Introduction

Most code coverage enforcers are tied to a specific code coverage tool like blanketjs or istanbul. This plugin works on the lcov file that can be generated by most test runners.

As a result you can work with any tests runners that are capable of generating the widely used LCOV file format.

Why is our plugin better

Most other coverage threshold tools fail the build only when you have a test and the test doesn't meet the threshold criteria. Most tools will actually pass the build if you dont write any tests for your project.

This plugin will check and fail if there are files that have not been covered at all.

You can always exclude the files that you specifically do not want to test for coverage. Since the policy is opt-out, less files will slip through the cracks ;-)

Getting Started

npm install grunt-code-coverage-enforcer --save-dev

Once the plugin has been installed, it may be enabled inside your Gruntfile with this line of JavaScript:

grunt.loadNpmTasks('grunt-code-coverage-enforcer');

The "code-coverage-enforcer" task

Enabling lcov reporter in your test runner.

You will need to enable lcov reporter in your test runner. The specific config for this may vary for each runner. Please check your specific runner for instructions.

Sample Intern config

intern:  {
   runner: {
        options: {
            config: "<intern- config>",
            runType: "runner",
            reporters: ["console", "lcov"]
        }
    }
  }

Sample Karma config

(http://karma-runner.github.io/0.8/config/coverage.html)

coverageReporter = {
  type : 'lcovonly',
  dir : 'coverage/',
  file : 'lcov.info'
}

Sample Istanbul config

(https://github.com/gotwarlost/istanbul, https://www.npmjs.org/package/grunt-istanbul-coverage, https://www.npmjs.org/package/grunt-mocha-istanbul)

Cli version of istanbul has a report parameter that you need to set to lcov or lcovonly. By default it should generate the lcov files. Most istanbul plugin allow this to be passed into their grunt config as well.

Here is one suc example for the grunt-mocha-istanbul plugin

  options: {
                    reportFormats: ['cobertura','lcovonly']
          }

Mocha

Mocha has an lcov reporter that can be foounf here https://nodejsmodules.org/pkg/mocha-lcov-reporter You can also use istanbul with Mocha using the plugin here https://github.com/pocesar/grunt-mocha-istanbul

This should typically generate an lcov.info file that will specified as input into the code-coverage-enforcer plugin.

Overview

In your project's Gruntfile, add a section named code-coverage-enforcer to the data object passed into grunt.initConfig().

grunt.initConfig({
  "code-coverage-enforcer": {
    options: {
        lcovfile: "lcov.info",
        lines: 60,
        functions: 60,
        branches: 60,
        src: "src",
        includes: ["src/**/*.js"],
        excludes: ["src/uncoveredfile.js"]
    }
  },
})

the default threshold values are

          {
            lines: 50,
            functions: 50,
            branches: 0,
            includes: ["**/*.js"],
            src: process.cwd(),
            excludes: [],
            logCurrentCoverage: false,//logs the coverage information for all the include files if true
            passMessage: "Yay! All is well!",//Message configuration for passed coverage checks
            failMessage: "Failed to meet code coverage threshold requirements.", //Message configuration for failed coverage checks.
            failBuild: true, //If you want to fail the build if coverage checks pass or fail.
            failBuildThreshold: 0 // Fine grained configuration to configure to build failures
        }

You can also provide multiple configurations for different features/packages of your project.

Options

lines

Type: Number Default value: 50

A Number value that is used to specify in % the line coverage required.

functions

Type: Number Default value: 50

A Number value that is used to specify in % the function coverage required.

branches

Type: Number Default value: 50

A Number value that is used to specify in % the branches coverage required.

src

Type: string Default value: src

The location of the folder within which we will search for files.

includes

Type: matcher Default value: *

The matcher required to compare files within the src location that are to be included

excludes

Type: [matcher] Default value: []

The matchers required to compare files within the src location that are to be excluded.

logCurrentCoverage

Type: boolean Default value: [false]

Can be used to log the current coverage limits.

passMessage

Type: string Default value: Yay! All is well!

Message to print when the coverage checks pass

failMessage

Type: string Default value: Failed to meet code coverage threshold requirements.

Message to print when the coverage checks fail

failBuild

Type: boolean Default value: true

Whether to fail the build when the coverage checks fail. Recommend to keep this configuration value to true always. You can set it to false, when you are initially trying/testing out this grunt task.

failBuildThreshold

Type: Number Default value: 0

This is a fine grained control that is added to control the behavior of the grunt task for existing projects that may not have good code coverage for all the files in the project. For e.g. in your project x if most of the files have coverage above 40%, but there are some files in the project whose coverage is below 40% and you expect that those files don't get the love they deserve, you can set the failBuildThreshold to 40%. This will put the files below 40% whose coverage dropped in the needs attention section and continue to fail build for the files whose coverage dropped for the files with existing coverage more than 40%. Needless to mention that the way the task would know the previous coverage configuration is if you tell this task what the existing coverage are by passing the information in 'src' option

extension

Type: regex Default value: /\.js$/

A regular expression used for matching the file extensions of source files that are to be included.

Usage Examples

Default Options

In this example, the default options are used

grunt.initConfig({
  "code-coverage-enforcer": {
    options: {
        lcovfile: "lcov.info",
        includes: ["src/**/*.js"],
        excludes: ["src/uncoveredfile.js"]
    }
  },
})

Custom Options

In this example, custom options are used to specify the threshold limits

grunt.initConfig({
  "code-coverage-enforcer": {
    options: {
        lcovfile: "lcov.info",
        lines: 60,
        functions: 60,
        branches: 60,
        src: "src",
        includes: ["src/**/*.js"],
        excludes: ["src/uncoveredfile.js"]
    }
  },
})

With the release of version 0.2.0, you will be able to customize the code-coverage on and individual folder/package level. The configuration options on an individual folder/package level can be specified in the following way:

grunt.initConfig({
  "code-coverage-enforcer": {
    options: {
        lcovfile: "lcov.info",
        lines: 60, //Global line coverage configuration
        functions: 60, // Global function coverage configuration
        branches: 60, // Global branch coverage configuration
        src: [{
          path:"middleware", 
          lines:90
        }, {
          path:"backend", 
          lines: 90, 
          functions: 80
        }, {
          path:"frontend", 
          lines: 90, 
          functions: 80,
          branches:70, 
          includes:["**.js"], 
          excludes:["frontend/exclude.js"]
        }
        includes: ["src/**/*.js"],
        excludes: ["src/uncoveredfile.js"]
    }
  },
})

In the above example we configured our project to have the following three coverage configuration

1. 90% line coverage, 60% function coverage, 60% branch coverage for all the files in middleware folder.
2. 90% line coverage, 80% function coverage, 60% branch coverage for all the files in backend folder.
3. 90% line coverage, 80% function coverage, 70% branch coverage for all the files in frontend folder except exclude.js file.

If any of the configuration option is not provided for a given folder/package, the default configurations are chosen.

Recommendations

We recommend using a specific task names build acceptance that will run the code-coverage-enforcer along with other build acceptance type of tasks.

Here is an example config that should go into your Gruntfile

    grunt.registerTask("build-acceptance", ["code-coverage-enforcer"]);

You can now simply invoke this from command line using

grunt build-acceptance

Contributing

The project includes jshint and jscs config as part of its dev grunt tasks. Please run grunt lint to ensure that the code passes the code conventions.

Once you have your code ready ready please make a pull request to the author.

Release History

March 17 2-15 0.3.0: Performance improvements, Refactoring of the code by using existing grunt tasks to simplify the code. Added a few more configuration options that will help with finer controls on how coverage enforcer functions.

Release History

March 17 2-15 0.2.0: Threshold configuration now supports per folder configuration in addition to one config for all.

June 19 2014 Initial Release.

License

Copyright (c) 2014 Intuit Inc.

Licensed under the MIT license.