@economist/backstopjs
v0.0.3
Published
BackstopJS: Catch CSS curveballs.
Downloads
29
Maintainers
Keywords
Readme
#BackstopJS Catch CSS curve balls.
BackstopJS automates visual regression testing of your responsive web UI by comparing DOM screenshots over time.
Features: Plays nice with multiple config files – Simulate user interactions with CasperJS scripts – Fast inline-CLI reports – detailed in-browser reports – CI Integration with JUnit reports – Test html5 elements like webfonts and flexbox – also plays nice with source control share your gold master with your team.
Version 2.0 is released!
$ npm install -g backstopjs
Here's whats new...
For 1.x users migrating to 2.x
- BackstopJS CLI can be installed globally (and it's recommended)
- All config paths are now relative to your current working directory
$(pwd)
. - There are new config properties
fileNameTemplate
Important if you are migrating 1.x configsid
Use this if you are sharing config files
Many many thanks for all who helped with this monumental task! 💙㊗️🙇 @JulienPradet, @onigoetz, @borys-rudenko, @ksushik, @dmitriyilchgmailcom, @Primajin
Contents
Tutorials, Extensions and more
##The BackstopJS workflow
Configure: Specify URLs, screen sizes, DOM selectors, ready events, interactions etc. (see examples directory)
Reference: Create a set of reference screenshots. BackstopJS will consider this your source of truth. (Update this whenever you want).
Test: BackstopJS creates a set of test screenshots and compares them with your reference screenshots. Any unwanted/unforeseen changes show up in a nice report.
##Getting started
Installation
Global installation (recommended)
Run this in your terminal from anywhere...
$ npm install -g backstopjs
Local installation (advanced)
Before installing locally, keep in mind that local installs do not put the Backstop
command on your application path. Please refer to the Local Install Usage section of the documentation.
To install locally, cd
into your project directory then...
npm install backstopjs
###Installing a development version
$ npm install -g garris/backstopjs#master
###Generating your configuration file
If you don't already have a BackstopJS config file. The following command will create a config template file which you can modify in your root directory. Note: this will overwrite any existing backstopjs config file.
From your projects's directory ...
$ backstop genConfig
By default, genConfig
will put backstop.json
at your current working path -- if you're not sure where this is, run echo $(pad)
, that's your current path. Also by default, a backstop_data
directory will be created at this same location.
The location of the backstop.json
file as well as all resource directories can be specified -- see Setting the config file path below.
###Working with your config file
####Here is the configuration that backstop genConfig
generates...
{
"id": "backstop_prod_test",
"viewports": [
{
"name": "phone",
"width": 320,
"height": 480
},
{
"name": "tablet_v",
"width": 568,
"height": 1024
},
{
"name": "tablet_h",
"width": 1024,
"height": 768
}
],
"scenarios": [
{
"label": "homepage",
"url": "https://garris.github.io/BackstopJS/",
"selectors": [
".jumbotron",
".row.firstPanel",
".firstPanel .col-sm-4:nth-of-type(2)",
".firstPanel .col-sm-4:nth-of-type(3)",
".firstPanel .col-sm-4:nth-of-type(4)",
".secondPanel",
".finalWords",
"footer"
],
"selectorExpansion": true,
"hideSelectors": [],
"removeSelectors": [],
"readyEvent": null,
"delay": 500,
"misMatchThreshold" : 0.1,
"onBeforeScript": "onBefore.js",
"onReadyScript": "onReady.js"
}
],
"paths": {
"bitmaps_reference": "backstop_data/bitmaps_reference",
"bitmaps_test": "backstop_data/bitmaps_test",
"casper_scripts": "backstop_data/casper_scripts",
"html_report": "backstop_data/html_report",
"ci_report": "backstop_data/ci_report"
},
"casperFlags": [],
"engine": "phantomjs",
"report": ["browser"],
"debug": false
}
####Required config properties
As a new user setting up tests for your project, you will be primarily concerned with these properties...
id
– The unique name of your config file. It's used by BackstopJS to manage and name files. It's useful to set this property for projects with multiple configs but it's required if you plan on sharing your reference files with teammates. If you're not sharing with others then you can omit this property -- BackstopJS will auto-generate one for you based on your file system.
viewports
– An array of screen size objects your DOM will be tested against. Add as many as you like -- but add at least one.
scenarios
– This is where you set up your actual tests. The important sub properties are...
scenarios[n].label
– Required. Used for screenshot naming.scenarios[n].url
– Required. Tells BackstopJS what endpoint/document you want to test. This can be an absolute URL or local to your current working directory.scenarios[n].selectors
– An array of CSS selector strings enabling you specify what part of your DOM you want to test. The default value isdocument
, which will attempt to capture your entire layout.
###Creating or updating reference bitmaps
From your project directory...
$ backstop reference
This will create a bitmaps_reference
directory with screen captures of all DOM elements specified in your config. See scenario filtering for more options.
###Generating test bitmaps
$ backstop test
This will create a new set of bitmaps in bitmaps_test/<timestamp>/
Once the test bitmaps are generated, a report comparing the most recent test bitmaps against the current reference bitmaps will run.
Significant differences will be detected and displayed in the browser report.
##Using BackstopJS
###Targeting elements
BackstopJS makes it super easy to capture screenshots of your entire layout or just parts of your layout. This is defined in the your scenario.selectors array. Each element of your array accepts standard CSS notation. By default BackstopJS takes a screenshot of the first occurance of any selector found in your DOM. e.g. If you have three li
tags in your layout only the first will used.
selectorExpansion
If you want BackstopJS to find and take screenshots of all matching selector instances then there is a handy switch for that...
scenarios: [
{
"selectorExpansion": true,
"selectors": [
".aListOfStuff li"
]
}
]
// captures all li children of the .aListOfStuff node
If you want very explicit controll of what you capture then you can disable selectorExpansion
and explictly select what you want...
scenarios: [
{
"selectorExpansion": false,
"selectors": [
".aListOfStuff li:nth-of-type(1)"
".aListOfStuff li:nth-of-type(2)"
".aListOfStuff li:nth-of-type(3)"
]
}
]
// does the same thing as above -- but more explicity.
###Incremental scenario reference/testing (filtering)
By default backstop.reference
will first remove all files in your reference directory then generate screenshots of all selectors specified in your config file.
If you need to update references (or test) for just one scenario you can do so by invoking BackstopJS with the --filter
argument...
$ backstop reference --filter=<scenario.label>
Aleternatively, if you don't want BackstopJS do first delete all files in your reference directory you can enable the incremental
flag.
$ backstop reference --i
Testing SPAs and AJAX content
It is very common for client-side web apps is to initially download a small chunk of bootstrapping code/content and render it to the screen as soon as it arrives at the browser. Once this has completed, various JS components often take over to progressively load more content.
The problem testing these scenarios is knowing when to take the screenshot. BackstopJS solves this problem with two config properties: readyEvent
and delay
.
NOTE: Advanced options also include very cool CasperJS features like waitForSelector() and waitUntilVisible() – see adding custom CasperJS scripts for more info...
####Trigger screen capture via console.log()
The readyEvent
property enables you to trigger the screen capture by logging a predefined string to the console. For example, the following line will delay screen capture until your web app calls console.log("backstopjs_ready")
...
"readyEvent": "backstopjs_ready"
In the above case it would be up to you to wait for all dependencies to complete before calling logging "backstopjs_ready"
string to the console.
####Delay screen capture
The delay
property enables you to pause screen capturing for a specified duration of time. This delay is applied after readyEvent
(if also applied).
"delay": 1000 //delay in ms
In the above case, BackstopJS would wait for one second before taking a screenshot.
In the following case, BackstopJS would wait for one second after the string backstopjs_ready
is logged to the console.
{
...
"readyEvent": "backstopjs_ready",
"delay": 1000 //delay in ms
}
Dealing with dynamic content
For obvious reasons, this screenshot approach is not optimal for testing live dynamic content. The best way to test a dynamic app would be to use a known static content data stub – or ideally many content stubs of varying lengths which, regardless of input length, should produce certain specific bitmap output.
Hiding selectors
That said, for a use case where you are testing a DOM with say an ad banner or a block of dynamic content which retains static dimensions, we have the hideSelectors
property in capture/config.json
which will set the corresponding DOM to visibility:hidden
, thus hiding the content from our Resemble.js analysis but retaining the original layout flow.
"hideSelectors": [
"#someFixedSizeDomSelector"
]
Removing selectors
There may also be elements which need to be completely removed during testing. For that we have removeSelectors
which removes them from the DOM before screenshots.
"removeSelectors": [
"#someUnpredictableSizedDomSelector"
]
Changing test sensitivity
"misMatchThreshold"
(percentage 0.00%-100.00%) will change the amount of difference BackstopJS will tolerate before marking a test screenshot as "failed". The default setting is 0.1
, this may need to be adjusted based on the kinds of testing you're doing.
More info on how misMatchThreshold is derrived can be found here... https://github.com/Huddle/Resemble.js/blob/af57cb2f4edfbe718d24b350b2be1d956b764298/resemble.js#L495
Capturing the entire document
BackstopJS recognizes a magic selector called document
. Use it to capture your entire HTML document (regardless of the height specified in your viewports
object).
"scenarios": [
{
"selectors": [
"document",
...
],
...
}
Note: This is required if you want to test an entire document layout with a height: 100%
rule specified on the <body>
element.
Testing across different environments
Comparing against different environments is easy. (e.g. compare a production environment against a staging environment).
To do this, add a referenceUrl
to your scenario configuration. When running $ backstop test
BackstopJS will use the url
for screen grabs. When running $ backstop reference
BackstopJS will check for referenceUrl
and use that if it's there. Otherwise it will use url
for both.
"scenarios": [
{
"label": "cat meme feed sanity check",
"url": "http://www.moreCatMemes.com",
"referenceUrl": "http://staging.moreCatMemes.com:81",
...
}
Running custom CasperJS scripts
Simulate user actions (click, scroll, hover, wait, etc.) or states (cookie values) by running your own Casper.js script on ready. For each scenario, the custom .js file you specify is imported and run when the BackstopJS ready event is fired.
From your project root, place your scripts in...
./backstop_data/casper_scripts
at the root of your config or in your scenario...
"onReadyScript": "filename.js" // Runs after onReady event on all scenarios -- use for simulating interactions (.js suffix is optional)
"onBeforeScript": "filename.js" // Runs before each scenario -- use for setting cookies or other env state (.js suffix is optional)
"scenarios": [
{
"label": "cat meme feed sanity check",
"onReadyScript": "filename.js" // If found will run instead of onReadyScript set at the root (.js suffix is optional)
"onBeforeScript": "filename.js" // If found will run instead of onBeforeScript at the root (.js suffix is optional)
...
}
Inside filename.js
, structure it like this:
module.exports = function(casper, scenario, vp) {
// scenario is the current scenario object being run from your backstop config
// vp is the current viewport object being run from your backstop config
// Example: setting cookies
casper.echo("Setting cookies");
casper.then(function(){
casper.page.addCookie({some: 'cookie'});
});
// `casper.thenOpen()` demonstrates a redirect to the original page with your new cookie value.
// this step is not required if used with _onBeforeScript_
casper.thenOpen(scenario.url);
// Example: Adding script delays to allow for things like CSS transitions to complete.
casper.echo( 'Clicking button' );
casper.click( '.toggle' );
casper.wait( 250 );
// Example: changing behavior based on config values
if (vp.name === 'phone') {
casper.echo( 'doing stuff for just phone viewport here' );
}
// ...do other cool stuff here, see Casperjs.org for a full API and many ideas.
}
Setting the base path for custom CasperJS scripts
By default the base path is a folder called scripts
inside your BackstopJS installation directory. You can override this by setting the paths.scripts
property in your backstop.json
file to point to somewhere in your project directory (recommended).
NOTE: SlimerJS currently requires an absolute path -- so be sure to include the full path when using the "engine": "slimer"
configuration option.
"paths": {
"casper_scripts": "backstop_data/scripts"
}
Reporting workflow tips
One testing approach to consider is incorporating BackstopJS into your build process and just let the CLI report run on each build. It's natural for your layout to break while you're in feature development -- refer back to the report when you feel things should be shaping up. Check the in-browser version of the report occasionally as needed when you need deeper information about what's happening in a test case.
CLI Report
Browser Report
Using the report property in backstop.json
enable or disable browser including/excluding the respective properties. E.G. The following settings will run both reports at the same time.
"report": ["browser", "CI"]
If you choose the CI-only reporting you can always enter the following command to see the latest test run report in the browser.
$ backstop openReport
Test report integration with a build system like Jenkins/Travis
The following config would enable the CI - report (default: junit format)
"report" : [ "CI" ],
The regression test report will be generated in the JUnit format and the report will be placed in the given directory (default: [backstopjs dir]/test/ci_report/xunit.xml).
You may customize the testsuite name and/or a report file (xunit.xml) path to your build report directory by using the below configuration overrides,
"paths": {
"ci_report" : "backstop_data/ci_report"
},
"ci": {
"format" : "junit" ,
"testReportFileName": "myproject-xunit", // in case if you want to override the default filename (xunit.xml)
"testSuiteName" : "backstopJS"
},
####CLI error handling
When a layout error is found in CLI mode, BackstopJS will let you know in a general report displayed in the console. In addition, BackstopJS will throw an error that will be passed to calling process.
###Using a js based config file
JSON-based configs cramping your style? Well, here's some good news -- BackstopJS allows you to import all config parameters as a node module (as an option instead of JSON) which allows you to use comments, variables and logic etc. inside of your config.
To use a js module based config file, explicitly specify your config filepath when running a command. e.g.
$ backstop test --configPath=backstopTests/someTest.js
See the next section for more info on setting the config file path.
Be sure to export your config object as a node module.
###Setting the config file path
Often, users have multiple config files to test various different scenarios or even different projects. By default, BackstopJS looks for backstop.json
in your project's root directory (in parallel with your node_modules
directory). You can override this by passing a --configPath
argument when running any command. e.g.
# example 1: run reference generation with absolute path
$ backstop reference --configPath=~/backstopTests/someTest.json
# Will capture reference files using scenarios from someTest.json inside backstopTests inside your home folder.
# example 2: run test with absolute path
$ backstop test --configPath=~/backstopTests/someTest.json
# Will run tests using scenarios from `someTest.json` inside `backstopTests` inside your home folder.
# example 3: run test with relative path
$ backstop test --configPath=backstopTests/someTest.json
# Will run tests using scenarios from `someTest.json` inside `backstopTests` inside your project root folder.
# example 4: run test with relative path and JS module
$ backstop test --configPath=backstopTests/someTest.js
# You can also specify your config parameters as a node module. This will import `someTest.js` from `backstopTests` inside your project root folder.
NOTES:
- all paths are relative to the location of the BackstopJS install directory (which is either inside your project's
node_modules
orbower_components
depending on how BackstopJS was installed). - Remember to add that extra
--
after thebackstop test
andbackstop reference
commands.
###Setting the bitmap and script directory paths
By default, BackstopJS saves generated resources into the backstop_data
directory in parallel with your backstop.json
config file. The location of the various resource types are configurable so they can easily be moved inside or outside your source control or file sharing environment. See below for the options...
Please note: these file paths are relative to your current working directory $(pwd).
...
"paths": {
"bitmaps_reference": "backstop_data/bitmaps_reference",
"bitmaps_test": "backstop_data/bitmaps_test",
"casper_scripts": "backstop_data/casper_scripts",
"html_report": "backstop_data/html_report",
"ci_report": "backstop_data/ci_report"
}
###Changing the rendering engine BackstopJS supports using PhantomJS or SlimerJS for web app rendering. (With thanks to CasperJS for doing the heavy lifting here.)
First, be sure to have SlimerJS installed. From your root directory run...
$ sudo npm install -g slimerjs
Then, in your backstop.json
config file, update the engine property to...
"engine": "slimerjs"
Thats it.
###Setting Casper command-line flags
This is for you if for some reason you find yourself needing advanced configuration access to CasperJS. You can set CasperJS flags via casperFlags
like so...
"casperFlags": [
"--engine=slimerjs",
"--proxy-type=http",
"--proxy=proxyIp:port",
"--proxy-auth=user:pass"
]
###Installing BackstopJS Locally The main reason to install backstop locally is likely to be a managed integration with a build implementation.
From the...
<your-project-path>/node_modules/backstopjs/
...directory you can run...
$ npm run reference
$ npm run test
$ npm run genConfig
$ npm run openReport
Which maps to the respective backstop <command>
.
Alternatively, when BackstopJS is installed locally, NPM will recognize the backstop <command>
pattern originating from your own NPM package.json
scripts. The following would enable you to run the
scripts: {
reference: backstop reference
test: backstop test
genConfig: backstop genConfig
}
The above is a crude example -- there are other fancy mappings you can create as well -- check out the NPM documentation for more info.
##Troubleshooting
###Migrating to 2.0
Filename issue: Projects don't work when I share with other users or run in different environments.
####If you just upgraded to 2.x from 1.x
Filename formats have changed. To use the 1.x (compatible) file format, use the fileNameTemplate
property like so...
{
...
fileNameTemplate: '{scenarioIndex}_{scenarioLabel}_{selectorIndex}_{selectorLabel}_{viewportIndex}_{viewportLabel}',
...
####If you are not migrating scripts but have recently upgraded BackstopJS
Be sure to use a config id
in your config file. See https://github.com/garris/BackstopJS/issues/291
###Windows users...
PhantomJS needs Python -- please make sure you have Python installed... see https://github.com/garris/BackstopJS/issues/185
###The dreaded command-not-found error...
Did you install BackstopJS with the global option? If installing globally remember to add that -g
when installing with npm i.e. npm install backstop -g
. If you installed locally, remember that the backstop <command>
pattern will only be available to your npm scripts -- see the local installation section above for more info.
###Debugging
To enable verbose console output when running your tests set the debug
property to true
in backstop.json
. This will also output your payload to the terminal so you can make sure to check that the server is sending what you expect. 😉
"debug": true
##Tutorials, Extensions and more
BackstopJS tutorial on css-tricks.com
A lovely article on Making Visual Regression Useful by Phillip Gourley
Automated regression testing for AngularJS (and other) web-apps -- article on DWB
Want to add BackstopJS to your existing gulp build? Turns out to be pretty easy – use gulp-chug. Learn how in this article by Filip Bartos.
Grunt fans -- check out grunt-backstop and this very nicely written article by Joe Watkins
Generate a BackstopJS configuration file from sitemap.xml with BackstopJS Scenarios Constructor by Enzo Sterro
BackstopJS brochure at http://BackstopJS.org/.
##Backstory BackstopJS is a useful wrapper around the very fabulous Resemble.js component written by James Cryer. Other implementations of Resemble.js, namely PhantomCSS require writing long form CasperJS tests -- which is of course great for testing complex UI interactions –- but kind of cumbersome for testing simple applications like static CMS templates, lots and lots of app states and different screen sizes.
BackstopJS may be just the thing if you develop custom WordPress, Drupal or other CMS templates. Tested on OSX.
BackstopJS was created by Garris Shipon during the Art.com labs years.
Follow @garris
##Gratitude Many many thanks to all the contributors with special thanks to...
- Suresh Kumar. M for help on the 1.3.2 release
- Klaus Bayrhammer for all the incredible effort leading up to 1.0 -- the cli reports and compatibility fixes are awesome!
- Evan Lovely and Klaus Bayrhammer for help on the 0.9.0 release
- Robert O'Rourke for help on the 0.8.0 release
- Klaus Bayrhammer for help on the 0.7.0 release
- Benedikt Rötsch for help on the 0.6.0 release
- Yulia Tsareva for help on the 0.5.0 release
- Lewis Nyman and Stoutie for help with 0.4.0 release -- you guys are responsible for really getting the ball rolling!
BackstopJS uses icons from the Noun Project