Yache

Cache package builds based on a hash from your source files, environment variables, and dependencies in yarn workspaces.

• Yache
  • Usage
    • Hooks
      • preCacheRestoreHook
      • cacheSavedHook
      • Example yache file
  • Problem
  • Development
    • Getting started
      • Dependencies
      • Integration tests.
      • Publishing

Usage

1. Add a yache.json file to each workspace package, which supports these properties:
   • hashExclude list of globstars, to exclude files from changing the cache of the build. EG src/**/*.test.ts would not affect your build output, so .test.ts files can be skipped when determining if source files have changed.
   • buildDirs directories to be cached, eg the build folder.
   • buildExclude globstars: If there are some files you don't want included, filter them out here. EG build/secret.pgp
   • buildCommand (optional) package.json script to run to build. Defaults to "build"
   • skipBuild (optional) if the module has no build process, it can be marked as skip.
   • env (optional) string array of environment variables that will affect the build output

{
  "/* in order to build this package yarn will use this script to build*/": "",
  "/* yarn run <buildCommand> */":"",
  "buildCommand": "build",

  "/* setting this to true indicates this package has no build step */": "",
  "skipBuild": false,

  "/* these are globstars for project files that should not affect the build cache */": "",
  "hashExclude": ["**/*.spec.*", "node_modules", "build"],

  "/* these are directories where builds should be zipped from */": "",
  "buildDirs": ["build", "../out"],

  "/* these are globstars for excluding files from within the build folders */": "",
  "buildExclude": ["**/.secret"],

  "/* these are environment variables that will affect the build */": "",
  "env": ["NODE_ENV"]
}

1. from the root of your workspace run yarn yache <app to build or restore cache>
   1. all package dependencies will also be built, or restored from cache.

Hooks

You can add a yache.ts to your workspace root.

preCacheRestoreHook

runs before a cache file is checked, and gives you an opportunity to restore a cache file from some long term storage.

/**
 * Use this to check s3 for a previous build with these source files.
 * @param fileName [string] tar file name
 */
export const preCacheRestoreHook = async (
  localFilePath: string,
  { cacheFileName }: Options
) => {
  try {
    await fs.access(localFilePath)
    console.error(`${cacheFileName} found locally`)
  } catch (e) {
    console.error(`Local cache miss "${cacheFileName}":`, e.message)
    await downloadS3File(cacheFileName)
  }
}

cacheSavedHook

runs after a cache file is generated, and gives you an opportunity to save a file to some long term storage.

/**
 *
 * @param fileName [string] filename to which was saved by yache
 */
export async function cacheSavedHook(fileName: string,  { cacheFileName }: Options) {
  const contents = await fs.readFile(fileName)
  await s3
    .upload({ Bucket: awsBaseConfig.bucket, Key: cacheFileName, Body: contents })
    .promise()
  console.error(`${cacheFileName} uploaded to s3`)
}

Example yache file

yache.ts

import * as AWS from 'aws-sdk'
import { spawnSync, spawn } from 'child_process'
import { join } from 'path'
import { promises as fs } from 'fs'
import { Options } from 'yache'

/*
  Setup environment variables for s3.
   ____  _____ _____ _   _ ____
  / ___|| ____|_   _| | | |  _ \
  \___ \|  _|   | | | | | | |_) |
   ___) | |___  | | | |_| |  __/
  |____/|_____| |_|  \___/|_|
*/

const awsBaseConfig = {
  bucket: process.env['ARTIFACT_BUCKET'],
  region: process.env['ARTIFACT_REGION'],
  accessKeyId: process.env['AWS_ACCESS_KEY_ID'],
  secretAccessKey: process.env['AWS_SECRET_ACCESS_KEY']
}

const s3 = new AWS.S3({ ...awsBaseConfig })

/*
  Yache Hooks
  These are called by yache, so that we can pull or push to s3 for persistent
  caches
  __   __         _            _   _             _
  \ \ / /_ _  ___| |__   ___  | | | | ___   ___ | | _____
   \ V / _` |/ __| '_ \ / _ \ | |_| |/ _ \ / _ \| |/ / __|
    | | (_| | (__| | | |  __/ |  _  | (_) | (_) |   <\__ \
    |_|\__,_|\___|_| |_|\___| |_| |_|\___/ \___/|_|\_\___/
*/

/**
 * Use this to check s3 for a previous build with these source files.
 * @param fileName [string] tar file name
 */
export const preCacheRestoreHook = async (
  localFilePath: string,
  { cacheFileName }: Options
) => {
  try {
    await fs.access(localFilePath)
    console.error(`${cacheFileName} found locally`)
  } catch (e) {
    console.error(`Local cache miss "${cacheFileName}":`, e.message)
    try {
      await downloadS3File(cacheFileName)
    } catch (e) {
      console.error('cache miss, waiting for yarn install\n', e.message)
    }
  }
}

/**
 *
 * @param fileName [string] filename to which was saved by yache
 */
export async function cacheSavedHook(fileName: string,  { cacheFileName }: Options) {
  const contents = await fs.readFile(fileName)
  await s3
    .upload({ Bucket: awsBaseConfig.bucket, Key: cacheFileName, Body: contents })
    .promise()
  console.error(`${cacheFileName} uploaded to s3`)
}

/*
 Utilities
 Utility functions used by the hooks.
   _   _ _   _ _   _ _
  | | | | |_(_) |_| (_) ___  ___
  | | | | __| | __| | |/ _ \/ __|
  | |_| | |_| | |_| | |  __/\__ \
   \___/ \__|_|\__|_|_|\___||___/
*/
const cachePath = join(__dirname, './.yache/')
/**
 * Download an s3 file and save it to .yache/
 * @param fileName [string] s3 file to look for
 */
async function downloadS3File(fileName: string) {
  const s3File = await s3
    .getObject({ Bucket: awsBaseConfig.bucket, Key: fileName })
    .promise()
  fs.writeFile(join(cachePath, fileName), s3File.Body)
  console.error(`${fileName} downloaded from s3`)
}

Problem

You can speed up build times of large projects by checking if any source files changed, and if they have not, reuse the build from last time. This is more complicated once you start splitting packages into modules. Consider the following simple example:

In this example, you may want to build package 1, which depends on package 3.

There are 4 scenarios:

1. pkg1 and pkg3 did not change.
   1. In this case, pkg1 build and pkg3 build cache can be used.
2. pkg1 changed, and pkg3 did not.
   1. pkg1 needs to be rebuilt after we use the pkg3 cache.
   2. pkg3 can use a cached build.
3. pkg3 changed, and pkg1 did not
   1. since pkg1 depends on pkg3, we should not assume that it's safe to use the previous build cache for pgk1.
   2. both packages should be rebuilt.
4. both packages changed 1.both packages need to be rebuilt

This demonstrates the complexities of trying to cache builds when files are split out, not to mention the complexities of trying to track package dependencies. Consider the following more complex example:

Lets break down just a single example of what needs to happen in order to build pkg1.

pkg1 directly depends on, pkg3, pkg4, and 5, and indirectly on 3, 4, 5, 6, 7, and 8. pkg3 depends on 5, 6, 7, pkg4 depends on 6 and 8.

If we want a built version of pkg1, and pkg 8 changed, that means a cache can be used for pkg3, 5, 6, and 7. And we need to rebuild pkg8, 4, and 1.

Manually writing a script to track this will get out of hand quickly, and is likely to get out of sync.

Development

.
├── dist                    # build directory
├── expectedOutput          # snapshots for integration tests
├── src                     # Modules
│   ├── cacheFileDefaults   # -> cache file writing and reading
│   ├── hashFS              # -> hashing files
│   ├── log                 # -> verbose logging utils
│   ├── merkle-tree         # -> Converts a package tree -> merkle tree, utils
│   └── package-tree        # -> reads workspace info into tree
└── test-app                # Test app
    └── packages
        ├── app-one
        ├── logger
        └── utils

Getting started

1. Run yarn from root
2. most dev work should be done in src
3. test-app is for integration tests. Run yarn integration-tests for integration test.

Each module under src has another README.md file, for more information

Dependencies

Docker, yarn, and node 12+ are required.

Integration tests.

yarn test will build a docker image, and run tests. See the Dockerfile.

Publishing

run yarn np and follow the prompts. Make sure you are following semver.

Currently the package is published by ericwooley, since reflektive doesn't have an npm account. If ericwooley no longer works at reflektive, you can't get ahold of him at: [email protected], and you need to publish, it's pretty trivial to change the name in the package.json, then install your package instead of the original.