Plumber

A Node-based tool for managing declarative web asset pipelines.

To install plumber, use npm to install the plumber-cli module (globally for ease of use):

$ sudo npm install -g plumber-cli

For an introduction, you may want to read the blog post Abstracting away the grunt work with Plumber.

Example

var all       = require('plumber-all');
var glob      = require('plumber-glob');
var bower     = require('plumber-bower');
var requirejs = require('plumber-requirejs');
var uglifyjs  = require('plumber-uglifyjs')();
var hash      = require('plumber-hash')();
var concat    = require('plumber-concat');
var less      = require('plumber-less')();
var filter    = require('plumber-filter');
var write     = require('plumber-write');

module.exports = function(pipelines) {

    var sources = glob.within('src');
    var writeToDist = write('dist');

    var requireConfig = {
        paths: {
            'event-emitter': '../../eventEmitter/EventEmitter'
        },
        shim: {
            'event-emitter': {exports: 'EventEmitter'}
        }
    };

    // Compile all JavaScript
    pipelines['compile:js'] = [
        all(
            sources('js/require.conf.js'),
            [sources('js/modules/app.js'), requirejs(requireConfig)],
            bower('underscore'),
            bower('pikaday', 'pikaday.js')
        ),
        uglifyjs,
        hash,
        writeToDist
    ];

    // Compile stylesheets
    pipelines['compile:css'] = [
        all(
            sources('stylesheets/reset.css'),
            [sources('stylesheets/less/*.less'), less],
            [bower('pikaday'), filter.type('css')]
        ),
        concat('style'),
        writeToDist
    ];

};

The Plumbing.js file above defines two sample pipelines:

• compile:js: Take all of the RequireJS config file, the main AMD app.js file compiled by RequireJS, the file exported by the underscore Bower component and the pikaday.js file in the pikaday Bower component, minimise all the JavaScript, hash the filenames and write the resulting JavaScript, sourcemaps and asset mapping files in the dist directory.

• compile:css: Take all of the reset.css file, the LESS files compiled to CSS, and the CSS files exported by the pikaday Bower component, concatenate them all into a single file named style.css and write the result in the dist directory.

You can run each individual pipeline with plumber <pipeline> or all of them with plumber.

Note: the syntax is still being defined and may change in the future.

Operations

Sourcing

• glob: find files using a path or pattern
• bower: find files from a Bower component
• lodash: generate a custom Lo-Dash build

Outputting

• write: write the result into files or directories
• s3: write files to Amazon S3

Compilation

• requirejs: compile an AMD module and its dependencies together
• uglifyjs: minimise JavaScript using UglifyJS
• less: compile LESS files to CSS
• libsass: compile Sass files to CSS
• myth: transform CSS files using the Myth preprocessor
• mincss: minimise CSS (using LESS)
• coffee: transpile CoffeeScript to JavaScript

Transformation

• rename: rename the filename of the input
• concat: concatenate all the input together
• filter: filter the input (e.g. based on file type)
• hash: hash the filenames and generate a mapping

Testing

• jshint: run JSHint on the input and produce a report

Meta-operation

• all: pass the input into the given set of operations and return the result

Principles

• Avoid boilerplate, use sensible defaults
• Hide operation internals behind a standard interface
• Make it trivial to write new operations
• Support outputing auxiliary files (sourcemaps, hash mapping, etc)
• Treat single-run and watch as different executions of a same defined pipeline
• Aim for high performance (exploit parallelism, caching, dirty-checking)
• Rely on typing of files to assert applicability of operations
• Allow specification of input files from config or as CLI arguments
• Support building assets and running linting/tests in the same way

Architecture

Most web asset building can be described as a pipeline of operations. Each operation takes one or more files as input and returns one or more files as output. The output of an operation can be piped as input to the next operation, creating a linear pipeline. Typically, source files are fed to the pipeline and the generated files are written to a destination directory.

This model works for a variety of file types (JavaScript, CoffeeScript, LESS, Sass, etc) and a variety of operations (minimize, transpile, AMD compilation, concatenation, etc). Linting and testing can even be modeled as a pipeline, where the output is the result or report.

An operation should only be concerned about doing a single thing well, and it is asynchronous by default using RxJS. Performance optimisation such as parallelism and caching are outside the scope of operations; instead, they are the sole concern of Plumber.

File data is currently being passed as strings, rather than streams, because most libraries that operations wrap do not support streams natively anyway...

Related projects

Grunt

The most popular task runner. Tasks are completely independent and executed imperatively.

Gulp and James

Both are stream-based pipelines of operations. The main difference with Plumber is the current lack of support for auxiliary files (e.g. sourcemaps) and the treatment of watch as a special listener which triggers a given block (e.g. re-run) on change.