Diff Dev Prod

Imagine the following scenario:

You are hired to build a simple static website, possibly using a static site generator.

However, the site is going to be hosted within some enterprise CMS, which was never intended to host static HTML.

So after creating your built files, you have to manually enter the information into the CMS, usually by copying and pasting small snippets of HTML into CMS components.

While this technically works as a way to host a website, the big problem with this setup (besides being cumbersome) is that the your source of truth is actually the CMS and not your source code, meaning it is possible to have your source code become out of sync with the CMS.

Once they become out of sync, determining what is different can be difficult for a variety of reasons:

• The CMS may insert various markup elements (e.g. HTML Comments, extraneous wrapper elements)
• The CMS may mangle or adjust your HTML
• You may not have a clean way to grab all the HTML from the CMS to start a comparison in the first place

Enter diff-dev-prod, or ddp for short. ddp helps you determine what the useful differences are between your development source code and the deployed production code.

Table of Contents

• Requirements
• Installation
• Usage
  • CLI Arguments and Options
  • Non-Interactive Mode Usage
  • Interactive Mode Usage
• Clean Configuration
  • Clean Configuration - JSON Structure
  • Clean Configuration - Inline JSON
  • Clean Configuration - File Path
  • Clean Configuration - Piped in via stdin
  • Clean Configuration - Motiviation / Example Scenario
• Example Output

Requirements

• GNU diff

This comes pre-installed with macOS and probably most linux/unix flavors.

To check this, run diff -v in your terminal. If you get a "command not found: diff" error, you'll need to install diff.

For example, on macOS, you can install this via homebrew:

brew install diffutils

Installation

To install the CLI globally, simply use npm or yarn via its respectful global flag:

npm install --global @designory/diff-dev-prod
# or
yarn global add @designory/diff-dev-prod

Alternatively, ddp can be run via npx or yarn dlx if you are using Yarn v2. Note that you'll need to pass in the full package name of @designory/diff-dev-prod when using npx or yarn dlx:

npx @designory/diff-dev-prod http://example.com
# or
yarn dlx @designory/diff-dev-prod http://example.com

Usage

ddp can be run in two modes: interactive and non-interactive mode. As the name implies, when in interactive mode, you can select the options you have not explicitly passed in.

By default, the program runs in non-interactive mode. To run the program in interactive mode, simply pass in the -i or --interactive flag.

Run ddp --help to view more information.

CLI Arguments and Options

$ ddp [options] <domain>

Domain Argument

The domain should be entered with the http:// or https:// protocol, and should not have an ending slash.

$ ddp http://example.com

Note that this argument is required unless in --interactive mode.

Options

Options / flags should be entered before the domain argument, and are detailed below:

-b, --build-dir=build-dir        The directory of your built HTML files. Defaults to "build".

-c, --clean-config=clean-config  A JSON string, or a file path to a JSON file, to configure how the
                                 HTML should be cleaned before it is diff'd. If "stdin" is passed,
                                 piped input will be used as the clean config. If invalid JSON is
                                 passed, the command exits early. See the DESCRIPTION section for
                                 more information on this.

-h, --help                       Show CLI help.

-i, --interactive                When true, will prompt for inputs it has not explicitly received.
                                 Defaults to false.

-o, --output=output              The filename of the HTML diff report. If "stdout" is passed, the
                                 unified diff string is echoed out.

-q, --quiet                      When true, suppresses any progress messages that otherwise would be
                                 logged out. Defaults to false.

-v, --version                    Show CLI version.

Non-Interactive Mode Usage

Copy a folder of your entire site and paste it in the current working directory. Then, run the following:

$ ddp http://example.com

This will compare your build against http://example.com.

Interactive Mode Usage

Let's say you have a website http://example.com and its contents should match up with a folder named build/ that looks like this:

build/
├── about/
│   └── index.html
├── contact/
│   └── index.html
└── index.html

After executing ddp --interactive, you might answer the prompts as:

$ ddp --interactive

? Enter the full root domain: https://example.com
? Enter the build folder path: build
? Enter the name of the output file: example.com.diff.html
? Select certain elements to clean: No
? Select certain attributes to clean: No

Requesting the following URLS:
http://example.com/
http://example.com/about/
http://example.com/contact/

Fetched 3 / 3
Done fetching URLs!
Preparing local files...
Preparing remote files...
Preparing to compute differences...
Computing differences...
Done computing all differences!
Written to example.com.diff.html

This creates your report in example.com.diff.html, ready to view the useful differences!

Clean Configuration

One of the most useful parts of ddp is the ability to clean the HTML before computing the differences.

HTML can be cleaned in two ways: by changing some elements, or by changing some attributes on an element.

When in interactive mode, you will be prompted for the configuration on how to clean the document. Otherwise, you must pass in a JSON configuration via the -c or --clean-config flag.

When using the --clean-config flag, the JSON can be passed in via three ways: inline JSON, a file path, or piped in via stdin.

Whichever way you pass in this configuration, it must be valid JSON and have a specific structure.

Clean Configuration - JSON Structure

When setting the clean config via JSON, the structure of the JSON should be a plain Object ({}), with two root keys: "elements" and "attributes". The values of each are an Array of plain Objects, detailed below:

{
    // An array of objects that configure which elements should be cleaned
    "elements": [
        {
            // A valid DOM selector for your element. This is required.
            "selector": "String",

            // After the elements have been selected, they can be filtered by
            // their contents for a simple string match.
            "contains": "String",

            // After the elements have been selected, they can be filtered by
            // checking their contents for a regular expression match.
            // Note that this regular expression should **not** contain starting
            // and ending slashes, and all backslashes should be escaped.
            // @example `"containsRegex": "jQuery v\\d+"`
            "containsRegex": "String",

            // When using the 'containsRegex' key, optional flags can be set via
            // the "containsRegexFlags" key.
            // @example `"containsRegexFlags": "i"`
            "containsRegexFlags": "String",

            // Enter true or false to flag whether the element should be removed
            // or not. If used with 'replacement', the entire element is replaced
            // with the passed HTML. This overrides 'empty' if both are set.
            "remove": true | false,

            // Enter true or false to flag whether the element should be emptied
            // or not (that is, have its contents removed). If used with
            // 'replacement', the element's contents are replaced with the
            // passed HTML. This is overridden by 'remove' if both are set.
            // If 'empty' or 'remove' are not set, the default action is to
            // empty the element.
            "empty": true | false,

            // If 'empty' or 'remove' are set, this value will be used to
            // replace the element or the element's contents. This value
            // will always be housed in an HTML comment.
            "replacement": "String",
        }
    ],

    // An array of objects that configure which attributes should be cleaned
    "attributes": [
        {
            // The attribute name to clean (e.g. "id"). This is required.
            "attribute": "String",

            // An optional selector, so this attribute will only be searched for
            // on this matching element.
            "selector": "String",

            // After the attribute has been selected, it can be filtered by
            // its value for a simple string match.
            "contains": "String",

            // After the attribute has been selected, it can be filtered by
            // checking its value for a regular expression match.
            // Note that this regular expression should **not** contain starting
            // and ending slashes, and all backslashes should be escaped.
            // @example `"containsRegex": "internal_id_\\d+"`
            "containsRegex": "String",

            // When using the 'containsRegex' key, optional flags can be set via
            // the "containsRegexFlags" key.
            // @example `"containsRegexFlags": "i"`
            "containsRegexFlags": "String",

            // Enter true or false to flag whether the attribute should be removed
            // or not. This is overridden by 'empty' if both are set. If 'empty'
            // or 'remove' are not set, the default action is to remove the attribute.
            "remove": true | false,

            // Enter true or false to flag whether the attribute should be emptied
            // or not (that is, have its value removed). If used with
            // 'replacement', the attribute's value is replaced with the
            // passed string. This overrides 'remove' if both are set.
            "empty": true | false,

            // If 'empty' is set, this value will be used to replace the attribute's value
            "replacement": "String",
        }
    ]
}

Clean Configuration - Inline JSON

If the clean configuriation is a JSON string, it is loaded directly.

ddp -c='{ "elements": [{ "selector": "head" }] }' http://example.com

Make sure any backslashes are escaped accordingly:

ddp -c='{ "elements": [{ "selector": "script", "containsRegex": "jQuery v\\d" }] }' http://example.com

Clean Configuration - File Path

If the clean configuration is a file, it is loaded and parsed as JSON.

// clean-config.json
{
    "elements": [
        { "selector": "head" }
    ]
}

$ ddp -c=clean-config.json http://example.com

Clean Configuration - Piped in via stdin

If the clean configuration is set to stdin, then the piped input will be parsed as JSON.

// clean-config.json
{
    "elements": [
        { "selector": "head" }
    ]
}

$ cat clean-config.json | ddp -c=stdin http://example.com

Clean Configuration - Motiviation / Example Scenario

Let's imagine your production site includes some analytics script tag at the bottom of the page. This tag is managed outside the static HTML you have built, and is injected at runtime by the production site.

When you are trying to determine the development vs. production differences, you already know that this script tag will be flagged as a difference. However, since the production site has a different mechanism for injecting the tag, this isn't a useful difference, and should be ignored.

To address this, we can clean both our local and production HTML before we compare the two.

Let's say your development HTML file looks like this at the end:

<!-- Local development HTML file -->
<body>
  ...
  <footer>Copyright 2020</footer>
</body>

And your production HTML file looks like this:

<!-- Remove production HTML file -->
<body>
  ...
  <footer>Copyright 2020</footer>
  <script>(function reportAnalytics() { /* ... */ })()</script>
</body>

To prevent that script tag from being shown as a difference, you'd use the following clean configuration:

// clean-config.json
{
    "elements": [
        {
            "selector": "script",
            "contains": "reportAnalytics",
            "remove": true
        }
    ]
}

Before starting the comparsion, both the local and remote HTML files will search for <script> tags that contain the string "reportAnalytics" and remove it. Since your local HTML file does not contain this tag, nothing changes. However, the HTML file for the remote production site will have this removed, thus allowing a more useful comparison.

Example Output

The output will be an HTML file that uses diff2html under the hood.

Open this file in your favorite browser to view the report.

The red (-) lines indicate what your local development build files show, and the green (+) lines indicate what your remote production pages show.

License

MIT

Author

Matt Wade