Dubstep

A batteries-included step runner library, suitable for creating migration tooling, codemods, scaffolding CLIs, etc.

Dubstep has utility functions for file system operations, Babel-based codemodding, Git operations and others.

License: MIT

Installation | Usage | API | Recipes | Motivation

Installation

yarn add @dubstep/core

Usage

import {
  Stepper,
  step,
  gitClone,
  findFiles,
  withTextFile,
  getRestorePoint,
  removeFile,
  createRestorePoint,
} from '@dubstep/core';
import inquirer from 'inquirer';

async function run() {
  const state = {name: ''};
  const stepper = new Stepper([
    step('name', async () => {
      state.name = await inquirer.prompt({message: 'Name:', type: 'input'});
    }),
    step('clone', async () => {
      gitClone('some-scaffold-template.git', state.name);
    }),
    step('customize', async () => {
      const files = await findFiles('**/*.js', f => /src/.test(f));
      for (const file of files) {
        withTextFile(file, text => text.replace(/{{name}}/g, state.name));
      }
    }),
  ]);
  stepper
    .run({from: await getRestorePoint(restoreFile)})
    .then(() => removeFile(reportFile))
    .catch(e => createRestorePoint(reportFile, e));
}
run();

API

Core | Utilities | top

All API entities are available as non-default import specifiers, e.g. import {Stepper} from '@dubstep/core';

Utilities can also be imported individually, e.g. import {findFiles} from '@dubstep/core/find-files';

Core

Stepper

import {Stepper} from '@dubstep/core';

class Stepper {
  constructor(preset: Preset)
  run(options: StepperOptions): Promise<any> // rejects w/ StepperError
  on(type: 'progress', handler: StepperEventHandler)
  off(type: 'progress', handler: StepperEventHandler)
}

type Preset = Array<Step>
type StepperOptions = ?{from: ?number, to: ?number}
type StepperEventHandler = ({index: number, total: number, step: string}) => void

A stepper can take a list of steps, run them in series and emit progress events.

step

import {step} from '@dubstep/core';

type step = (name: string, step: AsyncFunction) => Step;

type Step = {name: string, step: AsyncFunction};
type AsyncFunction = () => Promise<any>;

A step consists of a descriptive name and an async function.

StepperError

import {StepperError} from '@dubstep/core';

class StepperError extends Error {
  constructor(error: Error, step: string, index: number),
  step: string,
  index: number,
  message: string,
  stack: string,
}

A stepper error indicates what step failed. It can be used for resuming execution via restore points.

Utilities

File system | Babel | Git | Restore points | Misc

File system

findFiles

import {findFiles} from '@dubstep/core';

type findFiles = (glob?: string, filter?: string => boolean) => Promise<Array<string>>;

Resolves to a list of file names that match glob and match the condition from the filter function. Respects .gitignore.

moveFile

import {moveFile} from '@dubstep/core';

type moveFile = (oldName: string, newName: string) => Promise<any>;

Moves an existing file or directory to the location specified by newName. If the file specified by oldName doesn't exist, it no-ops.

readFile

import {readFile} from '@dubstep/core';

type readFile = (file: string) => Promise<string>;

Reads the specified file into a UTF-8 string. If the file doesn't exist, the function throws a ENOENT error.

removeFile

import {removeFile} from '@dubstep/core';

type removeFile = (file: string) => Promise<any>;

Removes the specified file. If the file doesn't exist, it no-ops.

withIgnoreFile

import {withIgnoreFile} from '@dubstep/core';

type withIgnoreFile = (file: string, fn: IgnoreFileMutation) => Promise<any>;
type IgnoreFileMutation = (data: Array<string>) => Promise<?Array<string>>;

Opens a file, parses each line into a string, and calls fn with the array of lines. Then, writes the return value or the array back into the file.

If the file does not exist, fn is called with an empty array, and the file is created (including missing directories).

withJsFile

import {withJsFile} from '@dubstep/core';

type withJsFile = (file: string, fn: JsFileMutation, options: ParserOptions) => Promise<any>;
type JsFileMutation = (program: BabelPath, file: string) => Promise<any>;
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

Opens a file, parses each line into a Babel BabelPath, and calls fn with BabelPath. Then, writes the modified AST back into the file.

If the file does not exist, fn is called with a empty program BabelPath, and the file is created (including missing directories).

See the Babel handbook for more information on BabelPath's API.

withJsFiles

import {withJsFiles} from '@dubstep/core';

type withJsFiles = (glob: string, fn: JsFileMutation, options: ParserOptions) => Promise<any>;
type JsFileMutation = (program: BabelPath, file: string) => Promise<any>;
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

Runs withJsFile only on files that match glob.

See the Babel handbook for more information on BabelPath's API.

withJsonFile

import {withJsonFile} from '@dubstep/core';

type withJsonFile = (file: string, fn: JsonFileMutation) => Promise<any>;
type JsonFileMutation = (data: any) => Promise<any>;

Opens a file, parses each line into a Javascript data structure, and calls fn with it. Then, writes the return value or modified data structure back into the file.

If the file does not exist, fn is called with an empty object, and the file is created (including missing directories).

withTextFile

import {withTextFile} from '@dubstep/core';

type withTextFile = (file: string, fn: TextFileMutation) => Promise<any>;
type TextFileMutation = (data: string) => Promise<?string>;

Opens a file, parses each line into a string, and calls fn with it. Then, writes the return value back into the file.

If the file does not exist, fn is called with an empty string, and the file is created.

writeFile

import {writeFile} from '@dubstep/core';

type writeFile = (file: string, data: string) => Promise<any>;

Writes data to file. If the file doesn't exist, it's created (including missing directories)

Babel

ensureJsImports

import {ensureJsImports} from '@dubstep/core';

type ensureJsImports = (path: BabelPath, code: string, options: ParserOptions) => Array<Object<string, string>>;
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

If an import declaration in code is missing in the program, it's added. If it's already present, specifiers are added if not present. Note that the BabelPath should be for a Program node, and that it is mutated in-place.

Returns a list of maps of specifier local names. The default specifier is bound to the key default.

If a specifier is already declared in path, but there's a conflicting specifier in code, the one in path is retained and returned in the output map. For example:

// default specifier is already declared as `a`, but trying to redeclare it as `foo`
ensureJsImports(parseJs(`import a from 'a';`), `import foo from 'a'`);
// > {default: 'a'};

A BabelPath can be obtained from withJsFile, withJsFiles or parseJs.

visitJsImport

import {visitJsImport} from '@dubstep/core';

type visitJsImport = (
  path: BabelPath,
  code: string,
  handler: (importPath: BabelPath, refPaths: Array<BabelPath>) => void),
  options: ParserOptions,
: void
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

This function is useful when applying codemods to specific modules which requires modifying the ast surrounding specific modules and their usage. This module works robustly across various styles of importing. For example:

visitJsImport(
  parseJs(`
    import {a} from 'a';
    a('test')
    console.log(a);
  `),
  `import {a} from 'a';`,
  (importPath, refPaths) => {
    // importPath corresponds to the ImportDeclaration from 'a';
    // refPaths is a list of BabelPaths corresponding to the usage of the a variable
  }
);

hasImport

import {hasImport} from '@dubstep/core';

type hasImport = (path: BabelPath<Program>, code: string, options: ParserOptions) => boolean
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

Checks if a given program node contains an import matching a string.

hasImport(
  parseJs(`
    import {a} from 'a';
    console.log(a);
  `),
  `import {a} from 'a';`
); // true

collapseImports

import {collapseImports} from '@dubstep/core';

type collapseImports = (path: BabelPath<Program>) => BabelPath<Program>

This function collapses multiple import declarations with the same source into a single import statement by combining the specifiers. For example:

import A, {B} from 'a';
import {C, D} from 'a';

// => 

import A, {B, C, D} from 'a';

generateJs

import {generateJs} from '@dubstep/core';

type generateJs = (path: BabelPath) => string;

Converts a Program BabelPath into a Javascript code string. A BabelPath can be obtained from withJsFile, withJsFiles or parseJs.

insertJsAfter

import {insertJsAfter} from '@dubstep/core';

type insertJsAfter = (path: BabelPath, target: string, code: string, wildcards: Array<string>, options: ParserOptions) => void
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

Inserts the statements in code after the target statement, transferring expressions contained in the wildcards list. Note that path should be a BabelPath to a Program node..

const path = parseJs(`const a = 1;`);
insertJsAfter(path, `const a = $VALUE`, `const b = 2;`, ['$VALUE']);

// before
const a = 1;

// after
const a = 1;
const b = 2;

It also supports spread wildcards:

const path = parseJs(`const a = f(1, 2, 3);`);
insertJsAfter(path, `const a = f(...$ARGS)`, `const b = 2;`, ['$ARGS']);

// before
const a = f(1, 2, 3);

// after
const a = f(1, 2, 3);
const b = 2;

insertJsBefore

import {insertJsBefore} from '@dubstep/core';

type insertJsBefore = (path: BabelPath, target: string, code: string, wildcards: Array<string>, options: ParserOptions) => void
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

Inserts the statements in code before the target statement, transferring expressions contained in the wildcards list. Note that path should be a BabelPath to a Program node..

const path = parseJs(`const a = 1;`);
insertJsBefore(path, `const a = $VALUE`, `const b = 2;`, ['$VALUE']);

// before
const a = 1;

// after
const b = 2;
const a = 1;

It also supports spread wildcards:

const path = parseJs(`const a = f(1, 2, 3);`);
insertJsBefore(path, `const a = f(...$ARGS)`, `const b = 2;`, ['$ARGS']);

// before
const a = f(1, 2, 3);

// after
const b = 2;
const a = f(1, 2, 3);

parseJs

import {parseJs} from '@dubstep/core';

type parseJs = (code: string, options: ParserOptions) => BabelPath;
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

Parses a Javascript code string into a BabelPath. The default mode is typescript. The parser configuration follows Postel's Law, i.e. it accepts all syntax options supported by Babel in order to maximize its versatility.

See the Babel handbook for more information on BabelPath's API.

parseStatement

import {parseStatement} from '@dubstep/core';

type parseStatement = (code: string, options: ParserOptions) => Node;
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

Parses a Javascript code statement into a Node. Similar to parseJs but extracts the statement node.

removeJsImports

import {removeJsImports} from '@dubstep/core';

type removeJsImports = (path: BabelPath, code: string, options: ParserOptions) => void
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

Removes the specifiers declared in code for the relevant source. If the import declaration no longer has specifiers after that, the declaration is also removed. Note that path should be a BabelPath for a Program node.

In addition, it removes all statements that reference the removed specifier local binding name.

A BabelPath can be obtained from withJsFile, withJsFiles or parseJs.

replaceJs

import {replaceJs} from '@dubstep/core';

type replaceJs = (path: BabelPath, source: string, target: string, wildcards: Array<string>, options: ParserOptions) => void;
type ParserOptions = ?{mode: ?('typescript' | 'flow')};

Replaces code matching source with the code in target, transferring expressions contained in the wildcards list. Note that path should be a BabelPath to a Program node.

replaceJs(
  parseJs(`complex.pattern('foo', () => 'user code')`),
  `complex.pattern('foo', $CALLBACK)`,
  `differentPattern($CALLBACK)`,
  ['$CALLBACK']
);

complex.pattern('foo', () => 'user code'); // before
differentPattern(() => 'user code'); // after

It also supports spread wildcards:

replaceJs(
  parseJs('foo.bar(1, 2, 3);'),
  `foo.bar(...$ARGS)`,
  `transformed(...$ARGS)`,
  ['$ARGS']
);

foo.bar(1, 2, 3); // before
transformed(1, 2, 3); // after

t

import {t} from '@dubstep/core';

t.arrayExpression([]);
// etc.

This is a flow-typed version of the exports from @babel/types. It is useful for creating AST nodes and asserting on existing AST nodes.

Git

gitClone

import {gitClone} from '@dubstep/core';

type gitClone = (repo: string, target: string) => Promise<any>;

Clones a repo into the target directory. If the directory exists, it no-ops.

gitCommit

import {gitCommit} from '@dubstep/core';

type gitCommit = (message: string) => Promise<any>;

Creates a local commit containing all modified files with the specified message (but does not push it to origin).

Restore points

createRestorePoint

import {createRestorePoint} from '@dubstep/core';

type createRestorePoint = (file: string, e: StepperError) => Promise<any>;

Creates a restore file that stores StepperError information.

getRestorePoint

import {getRestorePoint} from '@dubstep/core';

type getRestorePoint = (file: string) => Promise<number>;

Resolves to the index of the failing step recorded in a restore file.

Misc

exec

import {exec} from '@dubstep/core';

type exec = (command: string, options: Object) => Promise<string>;

Runs a CLI command in the shell and resolves to stdout output. Options provided are passed directly into execa.

Recipes

Preset composition

const migrateA = [
  step('foo', async () => gitClone('some-repo.git', 'my-thing')),
  step('bar', async () => moveFile('a', 'b')),
];

A task that needs to run the migrateA preset but also need to run a similar task migrateB could be expressed in terms of a new preset:

const migrateAll = [...migrateA, ...migrateB];

We retain full programmatic control over the steps, and can compose presets with a high level of granularity:

const migrateAndCommit = [
  ...migrateA,
  async () => gitCommit('migrate a'),
  ...migrateB,
  async () => gitCommit('migrate b'),
];

Restore points

If a step in a preset fails, it may be desirable to resume execution of the preset from the failing step (as opposed to restarting from scratch). Resuming a preset can be useful, for example, if a manual step is needed in the middle of a migration in order to unblock further steps.

const restoreFile = 'migration-report.json';
new Stepper([
  /* ... */
])
  .run({
    from: await getRestorePoint(restoreFile),
  })
  .then(() => removeFile(restoreFile), e => createRestorePoint(restoreFile, e));

Javascript codemods

// fix-health-path-check.js
export const fixHealthPathCheck = async () => {
  await withJsFiles(
    '.',
    f => f.match(/src\/.\*\.js/),
    path => {
      return replaceJs(path, `ctx.url === '/health'`, `ctx.path === '/health'`);
    }
  );
};

// index.js
import {fixHealthPathCheck} from './fix-health-path-check';
new Stepper([
  step('fix health path check', () => fixHealthPathCheck()),
  // ...
]).run();

Codemods with state

// fix-health-path-check.js
export const fixHealthPathCheck = async ({path}) => {
  const old = '/health';
  withJsFiles(
    '.',
    f => f.match(/src\/.*\.js/),
    path => {
      replaceJs(path, `ctx.url === '${old}'`, `ctx.path === '${path}'`);
    }
  );
  return old;
};

// index.js
import {fixHealthPathCheck} from './fix-health-path-check';
const state = {path: '', old: ''};
new Stepper([
  step('get path', async () => {
    state.path = await inquirer.prompt({
      message: 'Replace with what',
      type: 'input',
    });
  }),
  step('fix health path check', () => {
    state.old = await fixIt({path: state.path});
  }),
  step('show old', async () => {
    console.log(state.old);
  }),
]).run();

Leveraging Babel APIs (e.g. @babel/template)

import template from '@babel/template';

export const compatPluginRenderPageSkeleton = ({pageSkeletonConfig}) => {
  const build = template(`foo($VALUE)`);
  withJsFiles(
    '.',
    f => f.match(/src\/.\*\.js/),
    path => {
      path.traverse({
        FunctionExpression(path) {
          if (someCondition(path)) {
            path.replaceWith(build({VALUE: 1}));
          }
        },
      });
    }
  );
};

Complex state management

Since the core step runner library is agnostic of state, it's possible to use state management libraries like Redux to make complex state machines more maintainable, and to leverage the ecosystem for things like file persistence.

const rootReducer = (state, action) => ({
  who: action.type === 'IDENTIFY' ? action.who : state.who || '',
});
const store = redux.createStore(
  rootReducer,
  createPersistenceEnhancer('state.json') // save to disk on every action
);
new Stepper([
  step('who', async () => {
    const who = await inquirer.prompt({message: 'who?', type: 'input'});
    store.dispatch({type: 'IDENTIFY', who});
  }),
  step('resumable', async () => {
    const {who} = store.getState(); // restore state from disk if needed
    console.log(who);
  }),
]).run();

Motivation

Maintaining Javascript codebases at scale can present some unique challenges. In large enough organizations, it's not uncommon to have dozens or even hundreds of projects. Even if projects were diligently maintained, it's common for a major version bump in a dependency to require code migrations. Keeping a large number of projects up-to-date with security updates and minimizing waste of development time on repetitive, generalizable code migrations are just some of the ways Dubstep can help maintain high code quality and productivity in large organizations.

Dubstep aims to provide reusable well-tested utilities for file operations, Javascript codemodding and other tasks related to codebase migrations and upgrades. It can also be used to implement granular scaffolding CLIs.

Prior art

Dubstep aims to be a one-stop shop for generic codebase transformation tasks. It was designed by taking into consideration experience with the strengths and weaknesses of several existing tools.

Javascript-specific codemodding tools such as jscodeshift or babel-codemods can be limited when it comes to cross-file concerns and these tools don't provide adequate platforms for transformations that fall outside of the scope of Javascript parsing (for example, they can't handle JSON or .gitignore files).

Shell commands are often used for tasks involving heavy file manipulation, but large shell scripts typically suffer from poor portability/readability/testability. The Unix composition paradigm is also inefficient for composing Babel AST transformations.

Picking-and-choosing NPM packages can offer a myriad of functionality, but there's no cohesiveness between packages, and they often expose inconsistent API styles (e.g. callback-based). Dubstep can easily integrate with NPM packages by leveraging ES2017 async/await in its interfaces.

Why migration tooling

Generallly speaking, there are two schools of thought when it comes to maintaining years-long projects at large organizations.

The "don't fix what ain't broken" approach is a conservative risk management strategy. It has the benefit that it requires little maintenance effort from a project owner, but it has the downside that projects become a source of technology fragmentation over time. With this approach, all technical debt that accumulates over the years eventually needs to be paid in one big lump sum (i.e. an expensive rewrite).

Another downside is that this approach doesn't work well with a push maintenance model. Typically, dependencies in small projects are managed via a pull model, i.e. the project owner updates dependencies at their own convenience. However, in typical large cross-team monorepos, dependencies are managed via a push model, i.e. whoever makes a change to a library is responsible to rolling out version bumps and relevant codemods to all downstreams using that library.

The "always update" approach aims to keep codebases always running on the latest-and-greatest versions of their dependencies, and to reduce duplication of effort (e.g. in bug fixes across duplicated code or fragmented/similar technologies). This approach pays off technical debt incrementally, but consistently across codebases, with the help of tooling to ensure that quality in codebases remains high as improvements are made to upstream libraries (both as patches and breaking changes). The downside of this approach is that it requires a higher investment in terms of maintenance effort, but this is typically offset by offloading the cost of migrations/codemods to a platform/infrastructure team, rather than having every project team waste time on similar/repetitive manual migration tasks.

Regardless of which maintenance model an organization uses, migration tooling can be useful anywhere non-trivial improvements need to be made. Some examples include moving away from proprietary frameworks towards easier-to-hire-for open source ones, or moving away from undesirable technologies towards desirable ones (e.g. if a company decides to migrate from Angular 1.x to React for whatever reason).