@stackstorm/st2flow-yaml

npm i @stackstorm/st2flow-yaml -S

Critical Terminology:

• AST: Abstract Syntax Tree - a tree of tokens generated by a parser
• POJO: Plain Ol' JavaScript Object - the name says it all

This is a custom YAML parser, tokenizer, serializer, and AST manipulation utility. This module is comprised of the following sub-modules, which are described in further detail below:

• token-set.js - YAML parser and serializer, a thin wrapper around the popular yaml-ast-parser.
• crawler.js - utility for reading and writing to the AST. This is the primary liaison between the AST and business code and is built for consumption.
• (private) objectifier.js - internal utility for converting the AST to an easy-to-consume POJO.
• (private) token-factory.js - internal utility for creating AST tokens from raw JavaScript data.
• (private) token-refinery.js - internal utility for reindexing the AST whenever mutations are made.

NOTE: It will help to look at the actual code while you read along. Not all methods are documented - only the one's you should care about.

token-set.js

Given a YAML string, parses the string into an AST. This works by first parsing the string using the yaml-ast-parser. However, the yaml-ast-parser discards whitespace, comments, colons, and other special characters. To fix this, we recurse over the token tree and parse the "space in between" tokens. To illustrate this, consider the following yaml:

version: 1.0

# comment
description: >
  This is a description

Parsing the above YAML results in the following AST (trimmed for brevity):

{
  kind: 2,
  mappings: [{
  	kind: 1,
  	key: { startPosition: 0, endPosition: 7, kind: 0, value: 'version' },
  	value: { startPosition: 9, endPosition: 11, kind: 0, value: '1.0', valueObject: 1 }
  }, {
  	kind: 1,
  	key: { startPosition: 24, endPosition: 35, kind: 0, value: 'description' },
  	value: { startPosition: 41, endPosition: 62, kind: 0, value: 'This is a description' }
  }]
}

As you can see, there is a "gap" between a token's endPosition and the next token's startPosition. We manually parse this gap, assigning each token a "prefix": an array of tokens representing the "gap" before the respective token. This prefix is crucial for maintaining source identity so that the following is always true:

tokenSet.toYAML( tokenset.fromYAML(yaml_string) ) === yaml_string

TokenSet API and Usage

import { TokenSet } from '@stackstorm/st2flow-yaml';

const tokenSet = new TokenSet(yaml: string);

// parses the yaml string into an AST, replacing the previous AST
// this method is called during construction
tokenSet.fromYAML(yaml: string);

// converts the AST into a YAML string (this is where the prefix is crucial)
tokenSet.toYAML();

// converts the AST into a plain (easy-to-consume) JavaScript object
tokenSet.toObject();

// reindexes the AST after mutations are made
// consumers are responsible for calling this method after mutating the AST
tokenSet.refineTree();

crawler.js

This is a utility for reading and manipulating an AST. Consumers should never need to know how to read or mutate an AST directly. Instead, this module creates a "bridge" between the AST and business code by allowing consumers to interact with an AST as though it was a POJO. To illustrate this, consider the following YAML:

version: 1.0,
tasks:
  task1: 
  	 action: core.local cmd="echo 'Task 1'"
  	 next:
  	 	- do: task2
  	 	  when: <% succeeded() %>
  task2:
    action: core.local cmd="echo 'Task 2'"

The above YAML is represented as the following POJO, which is much easier to work with than the AST:

{
  version: 1,
  tasks: {
    task1: {
      action: 'core.local cmd="echo \'Task 1\'"',
      next: [
        { do: 'task2', when: '<% succeeded() %>' }
      ]
    },
    task2: {
      action: 'core.local cmd="echo \'Task 2\'"'
    }
  }
}

Whenever a mutation should be made to the AST, consumers can describe the mutation as if they were modifying the above POJO. This is explained in the next section.

crawler API and Usage

The crawler is a static objec for reading and mutating an AST. Each method expects an instance of TokenSet as the first parameter and a string path as the second parameter. The path uses deep.dot.syntax (or ['deep', 'dot', 'syntax']) to reference a key within a POJO. Continuing with the YAML/POJO example from above:

import { crawler } from '@stackstorm/st2flow-yaml';

crawler.getValueByKey(tokenSet, 'tasks.task2');
// -> { action: 'core.local cmd="echo \'Task 2\'"' }

// Array items should also use dot syntax, not bracket syntax
crawler.getValueByKey(tokenSet, 'tasks.task1.next.0.do');
// -> 'task2'

// Replace the value of a token - the value can be any data type
crawler.replaceTokenValue(tokenSet, 'tasks.task1.action', 'aws.lambda');

// Add a new key/value pair to an existing object - the value can be any data type
crawler.addMappingItem(tokenSet, 'tasks.task3', { action: 'core.local' });

// Remove a key/value pair from an object
crawler.deleteMappingItem(tokenSet, 'tasks.task3');

// Modify a collection using a technique analogous to `Array.prototype.splice`
crawler.spliceCollection(tokenSet, 'tasks.task1.next', 1, 0, { do: 'task3' });

objectifier.js (private)

This module is intended for internal use only and is not directly exposed to the outside world.

Given a token, recursively converts the token to a POJO for easy consumption. This is used in the TokenSet#toObject method to convert the AST into a plain object (see the crawler#getValueByKey method for more details). Most of the time you want to pass the root tree for a tokenSet:

Objectifier API and usage

import Objectifier from './objectifier';

const objectifier = new Objectifier(tokenSet.anchors);

objectifier.getTokenValue(tokenSet.tree);
// -> { version: 1, description: 'This is a description', ...}

token-factory.js (private)

This module is intended for internal use only and is not directly exposed to the outside world.

Given plain JavaScript data, creates the appropriate tokens for manipulating the AST. You can pass in any data type and expect the appropriate token in return. This is primarily used by the crawler for mutating the AST.

import factory from './token-factory';

factory.createToken('string');
// -> { kind: 0, value: 'string' }

factory.createToken(1234);
// -> { kind: 0, value: '1234', valueObject: 1234 }

factory.createToken(true);
// -> { kind: 0, value: 'true', valueObject: true }

factory.createToken({ an: 'object' });
// -> { kind: 2, mappings: [{ 
          kind: 1, 
          key: { kind: 0, value: 'an' },
          value: { kind: 0, value: 'object' }, 
        }]
      }

factory.createToken([ 'an', 'array' ]);
// -> { kind: 3, items: [
          { kind: 0, value: 'an' },
          { kind: 0, value: 'array' }, 
        ]
      }

token-refinery.js (private)

This module is intended for internal use only and is not directly exposed to the outside world.

Given the raw yaml data from the previous parsing, recursively "refines" all tokens in the tree, updating the startPosition, endPosition, jpath, and prefixes for all tokens. This utility should be used any time the AST is mutated and must be called manually. This is primarily used by the TokenSet#refineTree method to refine the entire AST on demand.

Recommended (indirect) use:

Call TokenSet#refineTree method (with no parameters) any time the AST is mutated:

import { crawler } from '@stackstorm/st2flow-yaml';

crawler.addMappingItem(tokenSet, 'tasks.task1', { action: 'core.local' });
tokenSet.refineTree();

Direct use:

The Refinery#refineTree method returns an object with the new tree and tail for a tokenSet instance. The TokenSet instance must be updated with the returned data.

Note: you should never have to do the following. It is only here for the sake of documentation. 99.99% of the time you will use the recommended technique above.

import { crawler } from '@stackstorm/st2flow-yaml';
import Refinery from './refinery';

crawler.addMappingItem(tokenSet, 'tasks.task1', { action: 'core.local' });
const refinery = new Refinery(tokenSet.yaml, tokenSet.head, tokenSet.tail);
const { tree, tail } = refinery.refineTree(tokenSet.tree);

tokenSet.tree = tree;
tokenSet.tail = tail;