@l.systems/config.io

v1.0.2

Published

3 years ago

configuration made easier

Downloads

0High
0Medium
0Low

even

config yaml extensible writable

Config.io

Config.io is a simple to use but very flexible configuration library based on YAML files.

Its main features are:

YAML based
writable in a monitorable way (changes are not saved in the main config file)
easily extensible format
supports import from other YAML files or JS (commonjs & ESM)

It is compatible from Node 10.x (only for its Common.JS version) and from 13.2 for its ESM version. Node 10 support will be dropped when it will loose its LTS status.

Note: The test suite needing the worker_threads module (ATM), it does not run on node 10.

Introduction

Though you should read the Getting started section to understand properly all concepts underlying this lib, you can skip reading to the Full examples if you prefer seeing sample use cases first and understand them properly later.

Topics

Getting started

As usual, install config.io through npm:

npm install @l.systems/config.io

Once it's done, you will usually need to create a config folder at the root of your project.

Core concepts of the library

There are 3 core concepts to understand to use this lib:

namespaces
environments
configuration tags

Namespaces are the basic scoping mechanism of config values inside config.io. They will be filled out in different files. You can deactivate namespaces by setting the simpleMode option to true (see later in the configuration to know how to achieve this). It will basically make the library hide the namespace concept and expose a single unnamed namespace (that will sometimes be reffered as the @ namespace) instead (though namespaces can still be used explicitely with the API).

Environments are sub units to namespaces. They allow you to easily change the configuration based on the context your program is executing. By default, config.io will load the following environments, in this order: 'default', process.env.NODE_ENV || 'development', 'local'. The later an environment is loaded, the more priority it has. If you have defined: port: 3000 in the default env and port: 80 in the local env, your app will get port: 80 when reading the configuration.

Configuration tags are custom tags exposed thanks to the YAML syntax being so powerful. By default, you get access to a collection of custom tags allowing you to create more complex configurations easily. You can deactivate tags you don't like, and you can also add tags that you need, the library being very extensible on those points. Here is an exemple so you can see what it can do:

port: !fallback
- !env NODE_PORT
- 3000

When accessing the port value, your app will read 3000 if the NODE_PORT environment variable is not set. Otherwise, it will get the NODE_PORT value.

Basic file naming scheme

The file naming conventions can be modified using the mapFilenamesTpl option that contains a lodash template string. It defaults to <%= ns || env %><% if (ns && env !== 'default') { %>.<%= env %><% } %> but you are free to change it as you wish.

That generates the following names for the files (for each environment):

# simpleMode = true (unnamed namespace)
[envName] -> [envName].yaml

# simpleMode = false
[ns] + default -> ns.yaml
[ns] + [otherEnv] -> ns.[otherEnv].yaml

Auto start

By default, config.io will try to be helpful and to load everything properly without you doing anything. It will "auto start" using default values.

The auto start mechanism is doing the following:

load the configio namespace explicitely
start the lib using the values in the namespace as its options

If you want to prevent the lib from doing an auto start, you can either:

set the CONFIGIO_NO_AUTO_START environment variable to any non empty value
pass the --configio-no-auto-start option to your program when starting it

If you prevent auto start, you will need to perform a manual start using the API.

Folder structure

There 2 important folders in the config.io library. The baseDir folder, and the configDir folder.

The baseDir is the folder from which every path in the configuration will be resolved. It defaults to a value that should be your project root. It can be provided either as a relative path from process.cwd() or as an absolute path. If not provided, config.io relies on the npm's find-root package. It basically search for the closest package.json in a folder stucture and is invoked like this: findRoot(process.argv[1]).

The configDir is the folder in which config.io looks for configuration files. It can be provided as either a relative path from baseDir or an absolute path. It defaults to config (the config folder in baseDir).

To explicitely set baseDir, you can either:

set the CONFIGIO_BASE_DIR environment variable
pass a --configio-base-dir option to your program
prevent config.io from doing an auto start (see above) and start the lib manually using the baseDir option

To explicitely set configDir, you can either:

set the CONFIGIO_DIR environment variable
pass a --configio-dir option to your program
prevent config.io from doing an auto start (see above) and start the lib manually using the configDir option

`null` and `undefined`

Good to know: YAML as no concept of undefined but its definition of null is nearly exactly the same as the definition of undefined in JS. So config.io always casts null values to undefined in JS and undefined values in JS as null in YAML.

That as the big advantadge that most validation libraries in JS do not treat null in a property the same way as undefined. If a property is optional, it can often be undefined but not null. This way, your lib will get undefined and not complain!

The big issue is that, for now, there is absolutely no way to get a null out of the config.io library if needed.

You can create pretty easily a !null custom tag to achieve this though. If it turns out it's a common use case, I'll gladly accept a PR on this.

Let's try it out!

Create a config folder with the following files:

configio.yaml:

autoload:
- server

server.yaml:

port: !fallback
- !env NODE_PORT
- 3000

Then create a test.js file in your project root looking like:

const configIO = require('@l.systems/config.io')

console.log(configIO.server.port)

And here are results from executing this very simple program:

node test.js
# 3000

NODE_ENV=8080 node test.js
# 8080

As you can see, your namespace values are readily available from the API, under a nested property. The name of the namespace in the library will be a camelCase version of the namespace name used in the configuration. Meaning you can safely name your namespaces using either - or _ like: server-plugins that will become serverPlugins in the JS API but will load the server-plugins.yaml and server-plugins.[env].yaml files.

If you are using the simpleMode, then the API will directly give you access to your namespace values. Eg:

config/configio.yaml:

simpleMode: true

config/default.yaml:

port: !fallback
- !env NODE_PORT
- 3000

./test.js:

const configIO = require('@l.systems/config.io')

console.log(configIO.port)

Will have the same effects as the program described earlier.

Note: You can safely use custom tags in the configio.yaml file too. And since the API loads the config.io files as a namespace, you can also change the configuration using environments.

Full examples

Those examples relies mostly on default values that should be fine for most projects.

With `simpleMode`

Let's say we want to make a very simple server to serve static files. We only need to get some basic server configuration, like port, the static folder in which to get the files, etc. The static folder in most use case should be looked into the dist folder of the project, because we compile them. But when developing, we'd like to serve the src folder instead. In production, we'd also like to be able to change the folder served using the --static option.

The simpleMode is very usefull for this use case, because it can limit the number of files we create.

Let's create the following folder structure:

package.json
config
- configio.yaml (it will contain options for config.io)
- default.yaml (it will contain default values for our app)
- development.yaml (it will contain specific dev values)
- local.yaml (it will contain specific local values, this file should usually not be commited)
app.js

config/configio.yaml

# we only need to activate simpleMode, other defaults are fine
simpleMode: true

config/default.yaml

# define good defaults
server:
  # use the --port option, fallback to NODE_PORT env var and then to 3000 if needed
  port: !fallback
  - !cmdline-option
    name: port
    type: number
    description: port on wich the server should listen to
  - !env NODE_PORT
  - 3000

  # more options here

# by default, the static folder is in
static: !fallback
- !cmdline-option
  name: static
  type: string
  description: path to the static folder to serve
- !path dist

config/development.yaml

# redefine specific values
static: !path src

config/local.yaml

# I'm lazy and on my computer, I don't like to use --port or NODE_PORT so I override port
server:
  port: 8080

app.js

const configIO = require('@l.systems/config.io')

// just for the example, get the values from the configuration
console.log(configIO.server.port, configIO.static)

And here are some results:

node app.js # 8080 /path/to/project/src
NODE_ENV=production node app.js # 8080 /path/to/project/dist
node app.js --static www # 8080 www

# let's remove config/local.yaml
rm config/local.yaml

node app.js # 3000 /path/to/project/src
NODE_PORT=8081 node app.js --port 8080 app.js # 8080 /path/to/project/src
NODE_PORT=8081 node app.js # 8081 /path/to/project/src

Without `simpleMode`

If we have a more involved application, where the server part is only one part out of many, we might find it cleaner to separate configuration values by namespaces. This will help finding them by separating concerns. The easier way to use those is either by autoloading them all, making them available as properties on the root object, or loading part of them using the !config custom tag.

In this example, we will use a little of both, so you can grab a complex case in one go.

We won't create environments for every namespace to keep it cleaner, but you obviously can.

Let's create the following structure:

package.json
config
- configio.yaml
- logger.yaml
- server.yaml
- plugin-static.yaml
- plugin-static.development.yaml
app.js

config/configio.yaml

autoload:
- server
- logger

logger.yaml

mode: detailed
transports:
  console: true
  logstash: http://url.to.logstash:port/

config/server.yaml

port: !fallback
- !env NODE_PORT
- 3000
plugins:
  static:
    bindTo:
      hostname: http://localhost:3000/
    options: !config plugin-static

config/plugin-static.yaml

chroot: true
addTrailingSlash: true
path: !fallback
- !cmdline-option
  name: static
  type: string
  description: path to the static folder to serve
- !path dist
# …

config/plugin-static.development.yaml

path: !path src

app.js

const configIO = require('@l.systems/config.io')

// just for the example, get the values from the configuration
console.log(
  configIO.logger.transports.logstash,
  configIO.server.port,
  configIO.plugins.static.options.path,
)

// the plugin-static namespace is also made available on the root object
console.log(configIO.pluginStatic.path)

Options

Here is a list of the options and their default values you can give the API to start it or set in your configio.yaml file:

{
  simpleMode: false,
  readOnly: true,
  autoSave: true,
  save: 'sync',
  allowAsync:
    yargs.argv.allowAsync ||
    !!process.env.CONFIGIO_ALLOW_ASYNC ||
    false,
  throwIfNotReady: true,
  customTags: {},
  mapFilenamesTpl:
    "<%= ns || env %><% if (ns && env !== 'default') { %>.<%= env %><% } %>"
  environments: [
    'default',
    process.env.NODE_ENV || 'development'
  ],
  inspectMode: 'detailed',
  resolve: {},
  autoload: [],
  baseDir:
    yargs.argv.configioBaseDir ||
    process.env.CONFIGIO_BASE_DIR ||
    findRoot(process.argv[1]),
  configDir: yargs.argv.configioDir ||
    process.env.CONFIGIO_DIR ||
    'config',
  logger: console,
}

options.simpleMode

boolean (defaults to false)

Allows to activate the simpleMode, very useful for simpler use cases.

options.readOnly

boolean (defaults to true)

Allows to deactivate read-only mode. Without setting this to false explicitely, you won't be able to make change to configuration values. Since it is an edge case, the library considers safer to treat configuration as read only by default.

options.autoSave

boolean (defaults to true)

If changes are allowed, should config.io try to save them automatically when you make one?

options.save

string (defaults to 'sync')

If changes are allowed, how should they be saved when save occurs. Available values are:

sync: save changes synchronously and immediately, very safe but slow
async: wait for no changes for 10 consecutive ms, then save
logger: don't actually save, log what would have been saved (using the debug() method of the logger)

options.allowAsync

boolean (defaults to false)

Should config.io allow namespaces to load values that need an async op to be available?

options.throwIfNotReady

boolean (defaults to true)

Throws if trying to access a namespace that has not loaded all its values yet. It obviously has no meaning if allowAsync is false.

Note: It is very dangerous to set this to false since you have no more garantees that you use namespaces safely. You might try to access a value that will be still undefined and set later by config.io.

options.customTags

{tagName: string}: boolean | customTag (defaults to {})

It allows you to (de)activate already existing custom tags (using a boolean). By default, all config.io tags are available as well as the YAML 1.1 timestamp tag.

config.io use the underlying yaml NPM package to parse YAML files. You should look there to have more info on other basic tags you can activate (like !!set, !!map, etc).

options.mapFilenamesTpl

string (defaults to "<%= ns || env %><% if (ns && env !== 'default') { %>.<%= env %><% } %>")

Allows to change the file scheme used by config.io. See above for more details.

options.environments

[string] (defaults to ['default', process.env.NODE_ENV || 'development'])

Changes the environments loaded by config.io. It always append 'local' last no matter what so don't put it in there.

options.inspectMode

string (defaults to 'detailed')

Changes how console.log() and more generally how util.inspect() behaves on configuration objects and arrays. Possible values are:

detailed: show how the values where constructed by displaying where custom tags where used and what their resolved values are
js: show how your program will see the configuration
debug: more useful for contributing to config.io than for using the lib, it shows internals

options.resolve

{namespace: string}: ({env: string}: string) (defaults to {})

It allows to change directories and names to load a specific env file. Eg:

resolve:
  # use @ to refer to the unnamed namespace in simpleMode
  @:
    # load development.yaml instead of test.yaml for env test
    test: development.yaml
  # write namespaces names here using camel case (same for envs below)
  nsName:
    # load the [nsName].yaml file from [configDir]/sub/dir
    default: sub/dir
    # load [configDir]/sub/dir/other.yaml instead of [configDir]/[nsName].test.yaml
    test: sub/dir/other.yaml

options.autoload

[string] (defaults to [])

Namespaces to make available automatically after auto start or manual start.

Note: this option is not allowed in simpleMode.

options.baseDir

string (defaults to yargs.argv.configioBaseDir || process.env.CONFIGIO_BASE_DIR || findRoot(process.argv[1]))

Change the base directory of config.io. It's not advised to set this option in configio.yaml file since either the file won't be found (since auto start won't be able to locate it) or it will only be active for subsequent loads, splitting namespaces folder between configio's namespace and the others.

options.configDir

string (defaults to yargs.argv.configioDir || process.env.CONFIGIO_DIR || 'config')

Change the config directory of config.io. It's not advised to set this option in configio.yaml file since either the file won't be found (since auto start won't be able to locate it) or it will only be active for subsequent loads, splitting namespaces folder between configio's namespace and the others.

options.logger

object { info: Function, debug: Function } | false (defaults to console)

Allows to change the default logger used by configio. Set to false to deactivate logging.

Note: logger can also be a function containing both an info and a debug property.

CommonJS and ESM

This package is available on both CommonJS & ESM versions. It uses the exports property of package.json to ease usage. So a simple:

import configIO from '@l.systems/config.io'

should load the ESM version of the package properly and lead to the same usage patterns as:

const configIO = require('@l.systems/config.io')

that should load the CJS version.

Though there are some tests ensuring that if both versions are used simultaneously everything works properly, important data being shared using a global safekeeped behind a Symbol property, it is still advised to stick with one version only (since some edge cases might arise).

Tags provided by default

Most of the following tags accept sub tags in some of their properties when they have a map form. That allows for more complex tags combinations.

!cmdline-option

This tag is useful to load data from the process' command line option into the configuration. It uses the yargs NPM package behind the scene to parse the arguments.

We strongly advise you to use the same lib if you want to support custom options of your own (not loaded in the configuration).

There are two ways to use this tag, as a scalar, or as a map:

# this will read the value of --some-option
someOption: !cmdline-option some-option
# this will read the value of --some-other-option but pass options to yargs to help it
# generate a beautiful --help message
someOpterOption: !cmdline-option
  name: some-other-option # accept sub tags
  type: string
  description: very useful option
  # …

!config

This tag allows you to load a namespace explicitely (so it works even in simpleMode). It only allows for a namespace name in scalar form (for now at least), like this:

# will load the server namespace and make it available in the server property of the config
server: !config server

Be careful, if the server namespace is loading async values, it will also make this one async!

!env

Just like !cmdline-option but for environment variables. It has also 2 forms, either a scalar or a map:

# set the value of the NODE_ENV environment variable in the mode property of the configuration
mode: !env NODE_ENV

# read the content of APP_DATA and try to parse it using JSON.parse()
# if parsing fails, use its string value instead
data: !env
  name: APP_DATA # accept sub tags
  json: true # json defaults to false

If you don't set json to true or use the scalar version, !env will try to cast the value as a Number. If the result of the cast is not NaN, it will use this value instead of the string. Eg:

port: !env NODE_PORT # returns 3000 and not '3000' for NODE_PORT=3000

!fallback

This tag allows you to consider multiple values in order and return the first non null / undefined one:

# Expose in port the value of the command line option --port
# if it's not set, fallback to the NODE_ENV environment variable
# if it's not set either, use 3000
port: !fallback
- !cmdline-option port
- !env NODE_PORT
- 3000

!import

This tag allows you to load data from external files. It currently support 4 loading mechanisms:

yaml
cjs (loading CommonJS files)
mjs (loading ESM modules)
js (loading CommonJS files by default, can be configured more)

Note: loading an ESM module causes an async op, thus to do this you need to have set the option allowAsync to true, otherwise it will fail to load the namespace.

You can use this tag in its simpler scalar form:

# supported file extensions:
# yaml, yml, json: use the yaml mode
# js, cjs: load CJS module (in this form, these are the same)
# mjs: load ESM module
import: !import some/file.cjs

Be careful, relative paths are resolved from baseDir and not configDir (this is usually more practical). You can also use absolute paths.

There is also a map form for this tag:

import:
  path: some/file.js # accept sub tags
  # type, string, only available if the file extension is '.js', can be either:
  # - (default) commonjs (use require() to load the file)
  # - module (use import() to load the file, careful: async op even for CJS modules!)
  type: commonjs
  # property, string, path to a property in the module to expose instead of the whole loaded value
  property: some.sub.prop # supports sub properties
  # throw, boolean (defaults to true), throws if imported value in null or undefined
  throw: true

!path

This tag allows to easily denote paths and handle their resolution from baseDir. It supports the 3 forms of YAML (scalar, map, seq):

# most basic usage, will resolve(baseDir, 'some/path')
simplePath: !path some/path

# map form, useful to use sub tags, here causing it to resolve(baseDir, process.env.STATIC_DIR)
staticFolder: !path
  path: !env STATIC_DIR

# seq form, joins every element in the array, supports sub tags
# this will result in: 'public/' + process.env.IMAGE_DIR_NAME
imageFolder:
- public/
- !env IMAGE_DIR_NAME

!regexp

This tags allows to easily load regexps from your configuration:

re: !regexp /test/g

You can also write regexp directly in your configuration using this (see Writing to the configuration for more on this).

!!timestamp

The YAML 1.1 !!timestamp tag is also available by default. It causes all strings looking like a date to be loaded as a Date() instead of a simple string.

# will create a Date() on 2020-01-12 at 00:00:00 UTC
# here is the spec for the allowed formats: https://yaml.org/type/timestamp.html
from: 2020-01-12

API

Starting `config.io`

If you disabled auto start, you can perform at any time a manual start. If you do a manual start after an auto start or another manual start, it will extend the global options internally and conform to simpleMode or autoload once again. You should avoid to switch the simpleMode value (except if auto start did nothing).

const configIO = require('@l.systems/config.io')

// performs a manual start
configIO({
  // options (see the options section)
})

// access your configuration values
console.log(configIO.server.port)

Waiting for async values

If you are using async values, you need to wait for the load to be done. You can do this by performing a manual start with no options (that will basically do nothing but return a promise on the previous start completion) or wait the result of your manual start:

const configIO = require('@l.systems/config.io')

const start = async () => {}
  // performs a manual start and wait for it to be done
  await configIO({
    // options (see the options section)
  })
  //- or wait for previous start to be ready
  await configIO()

  // access your configuration values
  console.log(configIO.server.port)
}

start()

Loading a single namespace explicitely

You can always load explicitely a namespace, even in simpleMode. Here is how to achieve this:

const configIO = require('@l.systems/config.io')

// ...
const namespace = /* await */ configIO('server')

console.log(namespace.port)
// ...

You can pass options to override defaults when loading a namespace:

configIO('server', options)

Options are basically the same as the global ones with the following differences:

simpleMode & autoLoad can't be provided
resolve takes only a {env: string}: string object since the namespace only needs to know how to resolve its own envirnoment files

And there are 2 specific options that are available only in this case:

prependEnvironments [string] (defaults to []): environments to load before the default ones
appendEnvironments [string] (defaults to []): environments to load after the default ones (but still before local)

Writing to the configuration

Basic

The most simple use case is just to save configuration changes that could be done maybe through, let's say, an API call:

const configIO = require('@l.systems/config.io')

// ...

const changePort = port =>
  configIO.server.port = port

And... That's it! config.io will take care to save the changes automatically. All changes are always made in the local environment. You cannot change other environments for safety reasons, and also to make changes easier to stop, since only changes will be visible in the local env and not all values loaded from other environments!

If you use autoSave: false, you need a way to save a namespace manually. That is achieved through the saveConfigIO() method, available on namespace's root:

// manually trigger a save (using the specified strategy in the 'save' option)
configIO.server.saveConfigIO()

Generate configuration files

To help you generate configuration files, the customTags classes used in the library are exposed. On recent node versions (supporting the exports property of package.json), you can access those using config.io/tags/tag-name[.mjs] modules.

Note: on older version of node, you will have to use the config.io/commonjs/tags/tag-name form to access those exports.

const configIO = require('@l.systems/config.io')
const { Env } = require('@l.systems/config.io/tags/fallback')
const { Fallback } = require('@l.systems/config.io/tags/fallback')

configIO.server.port = new Fallback([
  new Env({ name: 'NODE_PORT' }),
  3000,
])

We advise you to use autoSave: false to do this. Once you have completed the values you want in default.yaml, save, and rename the local file to the correct name for the default env of this namespace. If you want to generate more than one env for a configuration, you will need to load different instances of node for each one, since, for now, there is no way to force reload a namespace that is already loaded. We might implement this later.

Creating custom tags

You can easily create your own tags. For that, you can use either the YAML API or use the higher level createTag method exposed.

tags/null.js:

const { createTag } = require('@l.systems/config.io/create-tag') // careful, old node versions need commonjs/
const Null = module.exports.Null = class Null {
  valueOf() {
    return null
  }

  static yaml = {
    toYAML() {
      return null
    }
  }
}

module.exports.nullTag = createTag(Null)

config/configio.yaml:

customTags:
  null: !import
    path: ./tags/null.js
    property: nullTag

And you're done!

Using the library in a npm package

This is a very specific use case, but we wanted it to work! In this case, we strongly advise you to load your namespace explicitely and to provide your defaults through a prepended environment:

const configIO = require('@l.systems/config.io')
const path = require('path')

const myPkgConfig = configIO('my-pkg', {
  allowAsync: false,    // recommended
  prependedEnvironments: ['my-pkg-default-values'],
  resolve: {
    // you can change this path to whatever seems fine to you
    myPkgDefaultValues: path.join(__dirname, 'default-values.yaml'),
  }
})

We strongly advise you either not to accept async ops and thus to force it being false or to always await your config safely.

If you want to load internal values that should not be changed by the software you are using, you can have them in a separate, private, namespace (where you specify configDir when loading it so it cannot be looked for in the normal configuration folder of the main software).

Contributing

To get started, you need mercurial and the correct plugins. Here is how to install those:

# You obviously need to install pip before, on ubuntu run sudo apt install python-pip
pip install --user mercurial hg-evolve

Clone the repository and add in its .hg/hgrc:

[experimental]
evolution = all
topic-mode = enforce

[extensions]
evolve =
topic =

[hooks]
pretxncommit.prepublish = npm run prepublishOnly

Create a topic branch to host your code:

hg topic my-feature

For more on topic branches, read this tutorial.

Make your changes, commit as you wish, do your thing! Once you're ready to submit a change, request Developer access through Gitlab (you need an account). You should be granted some quickly.

Then push your topic branch to the server:

hg push -r my-feature

And open a merge request from it to branch/default!

That's it!

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

Config.io

Introduction

Topics

Getting started

Core concepts of the library

Basic file naming scheme

Auto start

Folder structure

null and undefined

Let's try it out!

Full examples

With simpleMode

Without simpleMode

Options

options.simpleMode

options.readOnly

options.autoSave

options.save

options.allowAsync

options.throwIfNotReady

options.customTags

options.mapFilenamesTpl

options.environments

options.inspectMode

options.resolve

options.autoload

options.baseDir

options.configDir

options.logger

CommonJS and ESM

Tags provided by default

!cmdline-option

!config

!env

!fallback

!import

!path

!regexp

!!timestamp

API

Starting config.io

Waiting for async values

Loading a single namespace explicitely

Writing to the configuration

Basic

Generate configuration files

Creating custom tags

Using the library in a npm package

Contributing

`null` and `undefined`

With `simpleMode`

Without `simpleMode`

Starting `config.io`