CloudConfig

Opinionated configuration and deployment for node.js projects on Google Cloud App Engine. Inspired by @invisible/deploy. Supports both dotenv and node-config for loading configuration.

Development is currently paused whilst we're working on a project with serverless architecture.

Latest npm package

Master branch

This is an alpha version still under heavy development. Some parts do not yet work and there might be breaking changes each patch version until v1.0.0. Dotenv and node-config haven't yet been wrapped to support transparent loading of config files from storage buckets alongside deployed config files in a running Google App Engine service instance. Files may still be locally synced and deployed alongside services.

Installation and use

Install Google Cloud tools ensuring gcloud and gsutil are on your path Copy Scripts into your package.json Run npm i @eyeswild/cloudconfig Run npm run setup:install

A rough guide to setting up your project

Create the following projects in Google Cloud Console:

• projectname, cloud storage for master config repository and localhost configuration buckets
• projectname-<environment>, for app configuration and running app service on the app engine instance, e.g.
  • projectname-development
  • projectname-staging
  • projectname-productionetc.

1. Check that you have write permissions to the new projects - billing must be enabled to allow creating storage buckets.
2. Copy scripts from below into your package.json, changing start scripts as needed for your app
3. Run npm run setup:storage to create the necessary minimum configuration for cloudconfig in your project
4. Add your app's configuration to the relevant dotenv .env or node-config .json files. storage/cloudconfig shouldn't contain any sensitive configuration as it is required for localhost use.
5. Search & replace to change projectname to your own app project name across all files in storage, and in cloudconfig.env
6. Add the storage folder to your .gitignore
7. Run a build then commit cloudconfig.env, and config folders
8. Run npm run setup:cloud to create config storage buckets in the correct cloud projects
9. Check that per-user permissions are correctly set on the storage buckets for whatever access control you desire for e.g. restricting developer access to production credentials

• To deploy to an environment, e.g. production:
  • Check that the App Engine configuration file for the environment is correct storage/cloudconfig/.env-<environment>.yaml
  • Run npm run cfg:push:production to sync config to the correct Google Cloud project and storage bucket for the environment
  • Run npm run deploy:production to create and deploy an instance of your app in that environment.

To support syncing a localhost configuration between developers: NOT YET FINISHED

1. Create a new service user for cloudconfig, e.g. [email protected]
2. Grant the service user Storage Legacy Bucket/Object Reader/Writer permissions as appropriate to enable pull/push of localhost configuration
3. Download or add service user credentials to storage/cloudconfig-service-account.json and commit
4. Other developers may update from repo and run npm run setup:developer

Push all configuration to repository

npm run cfg:push:master

Push relevant configuration to an environment

npm run cfg:push:<environment>, e.g: npm run cfg:push:development

Add config sync to your app code

import * as CloudConfig from '@eyeswild/cloudconfig'
import {createLogger} from 'winston'
import TODO from '@xdc/todo'

...

const log = createLogger({ ..... })    

...

// sync configuration overrides from appropriate google bucket 
// for project, environment, and service; for dotenv and 
// node-config configuration
CloudConfig.sync({
  logger: log,    // optional
  streamConfig: true
}).subscribe({
  next: (cfg: CloudConfig.Configuration) => {
    log.info('received configuration: ' + util.format(cfg))
  },
  complete: () => {
    logger.info('fetched app configuration, starting server...')
    runServer()
  },
  error: (err: any) => {
    logger.error(`whilst fetching app configuration - ${err}`)
    process.exit(1)
  },
})

function runServer() {
  TODO() // etc...
}

NPM 'cli' scripts for syncing configuration

A roll-your-own cli - just copy and paste these scripts into your package.json!

These scripts are cross-platform but note that npx and cross-env are required on the path

npm i npx cross-env -g

Other global dependencies are optional but recommended to speed up npx:

npm i copy env-cmd markcat trash-cli -g

"help": "npx markcat README-CLI.md",
"build": "tsc",
"clean": "npx trash compiled",
"empty-trash": "npx empty-trash",
"prestart": "npm run build",
"prebuild": "npm run clean",
"postbuild": "npm run cloudconfig:build",
"postclean": "npm run cloudconfig:clean",
"setup:storage": "npm i npx && npx copy \"./node_modules/@eyeswild/cloudconfig/storage/*\" ./storage && npx copy ./node_modules/@eyeswild/cloudconfig/cloudconfig.env . && mkdir env",
"setup:cloud": "npm run bucket:mk:root && npm run bucket:mk:all",
"setup:developer": "npm run setup:install && mkdir storage && mkdir env && npx env-cmd ./cloudconfig.env npm run cfg:pull:localhost",
"setup:install": "npm i -g npx cross-env env-cmd trash-cli empty-trash-cli copy && npm i && npm i -D",
"start": "npm run start:cloud",
"start:cloud": "node -r dotenv/config compiled/index.js dotenv_config_path=./dot.env",
"start:localhost": "npm run build && npx cross-env-shell NODE_ENV=localhost GOOGLE_APPLICATION_CREDENTIALS=storage/cloudconfig-service-account.json npm run start:cloud",
"app:deploy:production": "npx env-cmd ./storage/production/env/dot.env npm run util:app:deploy",
"cfg:push:all": "npm run cfg:push:development && npm run cfg:push:production && npm run cfg:push:master",
"cfg:push:localhost": "npx env-cmd ./storage/localhost/env/dot.env npm run util:cfg:push",
"cfg:push:development": "npx env-cmd ./storage/development/env/dot.env npm run util:cfg:push",
"cfg:push:production": "npx env-cmd ./storage/production/env/dot.env npm run util:cfg:push",
"cfg:push:master": "npx env-cmd ./cloudconfig.env npm run sync:push:root",
"cfg:pull": "npx cross-env-shell \"npm run sync:pull:services && npm run sync:pull:dotenv\"",
"cfg:pull:localhost": "npx env-cmd ./cloudconfig.env npm run util:pull:localhost",
"cfg:pull:master": "npx env-cmd ./cloudconfig.env npm run rsync:pull:root",
"cloudconfig:build": "npx copy \"./storage/cloudconfig/env-*\" . && npx copy \"./storage/cloudconfig/config/*\" ./config && npx copy \"./storage/localhost/env/*\" .",
"cloudconfig:clean": "npx trash env-*.yaml && npx trash config && npx trash dot.env",
"bucket:mk:all": "npm run bucket:mk:localhost && npm run bucket:mk:development && npm run bucket:mk:production",
"bucket:mk:localhost": "npx env-cmd ./storage/localhost/env/dot.env npm run util:bucket:create",
"bucket:mk:development": "npx env-cmd ./stoyrage/development/env/dot.env npm run util:bucket:create",
"bucket:mk:production": "npx env-cmd ./storage/production/env/dot.env npm run util:bucket:create",
"bucket:mk:root": "npx env-cmd ./cloudconfig.env npm run util:bucket:create",
"bucket:rm:all": "npm run bucket:rm:localhost && npm run bucket:rm:development && npm run bucket:rm:production",
"bucket:rm:localhost": "npx env-cmd ./storage/localhost/env/dot.env npm run util:bucket:remove",
"bucket:rm:development": "npx env-cmd ./storage/development/env/dot.env npm run util:bucket:remove",
"bucket:rm:production": "npx env-cmd ./storage/production/env/dot.env npm run util:bucket:remove",
"sync:push:services": "npx cross-env gsutil -m rsync -d -r ./storage/$NODE_ENV/default $CONFIG_BUCKET",
"sync:push:dotenv": "npx cross-env gsutil -m cp -r ./storage/$NODE_ENV/env/dot.env $CONFIG_BUCKET/env/dot.env",
"sync:push:root": "npx cross-env gsutil -m rsync -r ./storage $CONFIG_BUCKET",
"sync:pull:services": "npx cross-env gsutil -m rsync -d -r $CONFIG_BUCKET/config ./storage/$NODE_ENV/default/config",
"sync:pull:dotenv": "npx cross-env gsutil -m cp -r $CONFIG_BUCKET/env/dot.env ./storage/$NODE_ENV/env/dot.env",
"sync:pull:root": "npx cross-env gsutil -m rsync -r $CONFIG_BUCKET ./storage",
"util:pull:localhost": "npx cross-env-shell NODE_ENV=localhost CONFIG_BUCKET=$CONFIG_BUCKET-localhost npm run cfg:pull",
"util:bucket:create": "npx cross-env gsutil mb -p $GCLOUD_PROJECT $CONFIG_BUCKET",
"util:bucket:remove": "npx cross-env gsutil rb $CONFIG_BUCKET",
"util:app:deploy": "npx cross-env-shell \"npm run build && npm run util:deploy:gcloud\"",
"util:deploy:gcloud": "npx cross-env gcloud app deploy ./env-$NODE_ENV.yaml --project=$GCLOUD_PROJECT",
"util:cfg:push": "npm run sync:push:services && npm run sync:push:dotenv"

Documentation still "TODO":

• use with Compute Engine / non Google Cloud servers

Features still "TODO":

• stream configuration into wrapped node-config objects to support read only AppEngine file system
• move npm scripts into ccc.js cli
• support for creating multiple localhost hostname configurations for dev teams
• support service config versioning
• support splitting environments using services rather than projects
• ...
• cli tool to assist with manipulating and versioning configuration

License: MIT

Author: Michael Stuart-Matthews

Contributors: ___________________ <-- You