MonoRepository build tools

Mono-Repository -> M.R. -> Mr. -> Mister

mister is a set of commands that can super-simplify running tasks on a monorepository.

Mister...

• is a set of commands to simplify running tasks on packages in a monorepository.
• is not a build system or toolchain.
• does not require initialization, changes to your package.json.
• does not create symlinks, or magic fake relative packages with relative path imports, or anything else.
• may create one single file, .mister/build.json, to help prevent re-building packages that have not changed.

Finally, mister is not necessary. Everything mister does, you could do yourself, albeit much more tediously, if you use a similar folder structure.

mister may not be the right tool for you. There are plenty of monorepo systems available, from yarn workspaces, to lerna, and bit. Consult your programmers if mister runs longer than 4 hours.

What's so great about it

Mister tries to determine whether or not your task or command needs to be run. In general, if you build a package successfully, make no changes, and attempt to build it again, mister will skip that process. Tasks prefixed with an exclamation mark will skip this cache check and be run anyways- good for things like uni tests, integration tests, or acceptance testing.

mister do-all build !test
+[@scope/packace1:task:build] is up to date
+[@scope/package2:task:build] is up to date
+[@scope/package3:task:build] is up to date
-[@scope/package4:task:build] is out to date

mister do-all build !test
+[@scope/packace1:task:build] is up to date
+[@scope/package2:task:build] is up to date
+[@scope/package3:task:build] is up to date
+[@scope/package4:task:build] is up to date

This currently works by testing the timestamps of all the files in a package, minus those that are filtered through .gitignore (ideally, your source files). If you have a build task that generates files, and then you delete them yourself, mister won't try to rebuild them.

So how do I use it?

Mister's only real requirement is that your monorepository packages are in the folder packages/node_modules (although you can change it with the command line switch --package-prefix)

.
├── .mister             # You should .gitignore this if it gets created.
│   └── cache.json      # This is the file MR uses to keep track of build timestamps and dependencies.
├── node_modules        # This is where you put your external dependencies.
│   ├── express
│   └── underscore
└── packages
    └── node_modules    # These are the packages of your monorepository.
        ├── @scope
        │   ├── package1
        │   └── package2
        ├── package3
        └── package4

Now, you can have mister do some things for you:

mister do @scope/package4 --tasks clean !test build

Or just do some things on all your packages, if you keep common task names:

mister do-all build !test

How do I make package bundles? (TGZ)

mister comes with a pack command, which honors bundledDependencies.

mister pack @scope/package2 package1

Note that due to npm pack not following symlinks or doing any module resolution at all (it assumes everything is in relative node_modules), mister will perform npm install --production for all local package dependencies, and rename package-local dependency versions in package.json with the relative file path to the dependency tarball. Upon completion, all changes are reverted (the only side effects are tarballs in ./dist).

This process can be time-consuming.

mister pack is a primitive that only creates packages. You will need to build them first e.g. mister do packageName --tasks build && mister pack packageName

I want to share code between AWS functions easily!

mister comes with a zip command, which honors bundledDependencies. This is why I made mister:

tree
.
├── .mister
│   └── cache.json
├── node_modules
│   ├── redis
│   └── pg-promise
└── packages
    └── node_modules    # These are the packages of your monorepository.
        ├── @lambda
        │   ├── someLambdaFunction
        │   └── someOtherLambdaFunction
        ├── @shared
        │   ├── some-useful-function
        │   └── some-shared-utility

mister do-all build !test && mister zip

tree
.
├── .mister
│   └── cache.json
├── dist
│   ├── lambda-some-lambda-function-1.0.0.zip # Here is my first function
│   └── some-shared-utility-1.0.0.zip         # here is my second function
├── node_modules
( continues... )

Both functions have redis, pg-promise, @shared/some-useful-function, and @shared/some-shared-utility in bundledDependencies with all necessary dependencies, and are bundled in a zip with package.json at the top level, ready for direct deployment to AWS using your method of choice.

Reference

Any other recommendations?

To take advantage of not having multiple dependency versions, you'll want to install all your dependencies at the top level (but you will still need to reference those dependencies in the package.json files of your packages). But you don't have to do that if you don't want to. Managing your dependencies is up to you ;)

How does this even work?

It helps to read up on how node resolves module names. The TLDR version is that, given require('module-name'), node will essentially walk up the filesystem until it discoveres a path matching node_modules/module-name.

Using package/node_modules to structure your monorepository means you can leverage this to require('your-monorepository-package') from inside of any other monorepository package, and also let you require('external-dependency') installed to your top-level node_modules folder.

Caveats

node_modules is Ignored

Several tools, like ts-node, by default ignore node_modules, which means they do not work as expected in packages/node_modules. If you have written tests in typescript, you will need to pass an updated TS_NODE_IGNORE environment variable either through your top-level script or your package-level script:

{
    "scripts": {
        "test": "cross-env TS_NODE_IGNORE=\"/(?<!packages\/)node_modules/\" nyc mocha",
    }
}

github pull requests and diffs say changes in my packages are "binary" and won't show them.

Just like above, GitHub treats node_modules as a magical string. But for now you can click on the fake greyed-out files to show the diff.

Some npm scripts fail because they can't find installed bins.

If you cwd into a package directory, npm will add ./node_modules/.bin to PATH at runtime to try to run things there. You can use PATH=$PATH:../../node_modules/.bin npm run script-name as a shortcut. Many editors also have an 'Open In Terminal': Visual Studio Code has a integrated.terminal.env option to allow you to make that path explicit to the workspace root.

./vscode/settings.json

{
    "terminal.integrated.env.windows": {
        "Path": "${workspaceRoot}\\node_modules\\.bin;${env:Path}"
    },
    "terminal.integrated.env.linux": {
        "PATH": "${workspaceRoot}/node_modules/.bin:${env:PATH}"
    },
    "terminal.integrated.env.osx": {
        "PATH": "${workspaceRoot}/node_modules/.bin:${env:PATH}"
    }
}

Incorrect Paths

A number of modules, like app-root-dir, make assumptions about where they are located and can return incorrect path results.

Greedy Dependencies

Even worse, some like uglifyjs-webpack-plugin can break your builds because they will write files assuming they have ownership of the package-local node_modules entry, creating directories and files that don't actually exist, and breaking subsequent runs.

Given this simplified structure:

cwd = ./packages/node_modules/monorepo-package
.
├── node_modules
│   └── uglifyjs-webpack-plugin         # The first time, require('uglifyjs-webpack-plugin') resolves here
│       └── index.js
└── packages
    └── node_modules
        └── monorepo-package

Running webpack will create the file ./packages/node_modules/monorepo-package/node_modules/uglify-webpack-plugin/.cache

cwd = ./packages/node_modules/monorepo-package
.
├── node_modules
│   └── uglifyjs-webpack-plugin             # This is where the module really is!
│       └── index.js
└── packages
    └── node_modules
        └── monorepo-package
            └── node_modules
                └── uglifyjs-webpack-plugin  # but require('uglifyjs-webpack-plugin') will now resolve this!!
                    └── .cache

The next time you build, node will resolve require('uglify-webpack-plugin') to that that folder instead. That folder only has the cache file, the require will throw an error because it cannot find any exported function, and your build will fail.

Thanks

A huge thanks to @a-z for agreeing to free up the name mister.