Revibe - JSON templating and preprocessing

• Example

• Installation

• Documentation

  • Overview
  • Key prefixes
  • Transformers
    • File
    • Shell
    • Http
    • Javascript

• Missing/Planned

• Usage considerations

• Secuirty considerations

Example

consider the following setting:

question.txt

What is best in life ?

answers.txt

Raindrops on roses
Whiskers on kittens
Crushing your enemies, seeing them driven before you and hearing the lamentations of their women

answers.json

[
	"Raindrops on roses",
	"Whiskers on kittens",
	"Crushing your enemies, seeing them driven before you and hearing the lamentations of their women"
]

demo.rvb

This is an example of a revibe template. A revibe template is itself a JSON document whose keys and values can be either carried as-is into the resulting JSON or can be interpreted as revibe expressions by one of the defined transformers allowing data to be fetched from various sources and embedded into the resulting document.

{
    "//!":"This is a comment, it will not be in the result",
    "Hello":"World",
    "//!2":"This is also comment, it is in the same object as the first comment so it needs a unique key",
    "//!3":"",
    "//!4":"",
    "//!5":"Let see some transforms:",
    "TransformExamples":
    {
        "Shell":
        {
            "//!":"Execute a shell command and use it's result",
            "!HelloWorldFromShell":"shell://echo 'Hello world'"
        },
        "Javascript":
        {
            "//!":"Evaluate a javascript expression and use it's return value",
            "!GeneratedOn":"js:// new Date()"
        },
        "File":
        {
            "//!1":"Read a text a file in a shell/os agnostic way",
            "!Question":"file://question.txt",
            "//!2":"Read a json file as text",
            "!AnswerJson":"file://answer.json",
            "//!3":"Read a json file and compose into result",
            "!!Answer":"file://answer.json"
        }
    }
}

Revibe templates can be transformed via the rvb command line program

$ rvb -i demo.rvb

with the result going to standard output

{
    "Hello": "World",
    "TransformExamples": {
        "Javascript": {
            "GeneratedOn": "2016-07-01T16:06:45.585Z"
        },
        "Shell": {
            "HelloWorldFromShell": "Hello world"
        },
        "File": {
            "Question": "What is best in life ?",
            "AnswerFromText": [
                "Raindrops on roses",
                "Whiskers on kittens",
                "Crushing your enemies, seeing them driven before you and hearing the lamentations of their women"
            ],
            "AnswerFromJson": [
                "Raindrops on roses",
                "Whiskers on kittens",
                "Crushing your enemies, seeing them driven before you and hearing the lamentations of their women"
            ]
        }
    }
}

or in node via the revibe module

require('revibe')
.Revibe(
    JSON.parse(fs.readFileSync('demo.rvb'))
)
.then(console.log)

With the result being provied as javascript object to the provided continuation, in this case console.log.

Revibe uses ES6 Promises and targets node 4.2 and above.

Installation

OSX and Linux

From npm registry

$npm install revibe

Via npm without the registry

Download the latest .tgz package from the releases section, unpack to a local directory, change to it and run

$npm install .

Without npm

Download the latest .tgz package from the releases section, unpack to a local directory and optionally add it to your $PATH

Windows

TBD

Documentation

Key prefixes

A key prefix indicates how the corresponding value will be interpreted and embedded into the resulting document, All revibe prefixes end with a ! character, property names that are not prefixed are transformed recursively as templates.

The following key prefixes are supported:

//! - Comment, both the key and the value will not be present in the output

! - The value returned by a transformer will be used as-is (string for shell, file and http transformers and according to the type returned by the javascript expression for the javascript transformer)

*! - The value returned by a transformer is a multi-line text that will be split to lines and embedded as an array of strings

!! - The value returned is a JSON text which will then be parsed and embedded in the resulting document as an object

!#! - The value returned is a JSON text representing a revibe template which will then itself be transformed by revibe and the result of the transformation embedded in the resulting document.

Transformers

File

Reads the content of a file from the filesystem.

Shorthand expression

file://<path>

where path is either an absolute or a relative (to cwd) path

Object notation

{
    "Type":"file",
    "Expression": <path>
    "TrimTrailingNewline": <boolean> // Default: true
}

Shell

Executes a command in user's default shell and returns the output

Shorthand expression

shell://<cmd>

where cmd is the command to execute

Object notation

{
    "Type":"shell",
    "Expression": <path>
    "TrimTrailingNewline": <boolean> // Default: true
}

Quotes and shell variable expansion

Because the shell expressions are enclosed in JSON string literals double quotes (") need to be escaped in shell commands to allow for shell variable expansion:

{"!path":"shell:// echo 'The current PATH environment variable is $PATH'"}

will transform to

{"path": "The current PATH environment variable is $PATH"}

whereas

{"!path":"shell:// echo \"The current PATH environment variable is $PATH\""}

will transform to

{ "path": "The current PATH environment variable is /Library/Frameworks/Python.framework/Versions/3.5/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/local/share/dotnet/bin:/usr/local/share/dotnet:/usr/local/git/bin:/usr/local/go/bin" }

or whatever your $PATH is.

*** Multiline split ***

Multiline split is a key prefix that is available for all transformers and is particularly useful with shell commands. e.g.:

{"*!files":"shell//ls -1"}

will transform nicely into

{"files":["file1","file2",...,"fileN"]}

Http

Fetches a web resource

Shorthand expression

URL

Object notation

{
    "Type":"http", // also for https
    "Expression": <url>
    "Headers": {
        "User-Agent","revibe"
    }
}

Currently only the GET method is suported

Javascript

Evaluates a javascript expression and returns the result

Shorthand expression

js:// <js code>

Object notation

{
    "Type":"js",
    "Expression": <js code>
}

Missing/Planned

• More features for the file:// transformer

  • Handle groups of files sensibly, such as "file://*.json" - merge into same
    object/object-per file/array ?

• js transformer - detect when the returned value is a promise and use it's result.

• In rvb (command line tool) run a user-provided js code to prep the node environment for stuff used by "js://" expressions prior to transform.

• A key prefix to have resulting JSON fields replace the fields in the parent scope

{"<!!foo":"{'bar':'baz'}"} => {"bar":"baz"}

vs

{"!!foo":"{'bar':'baz'}"} => {"foo":{"bar":"baz"}}

vs

{"!foo":"{'bar':'baz'}"} => {"foo":{"{'bar':'baz'}"}}

• Better JSON parsing and serialization:

  • Deterministic serialization - preserve template order for emitted properties.
  • Better error reporting (where is that goddamn breaking trailing coma ?)

• Extensibility for pluggable third-party transformers (mongo, couch, elasticsearch etc.)

• Browser use case and relevant transformers (XHR for http etc.)

Usage considerations

Revibe is meant to make the creation and management of JSON documents used in devops tools, configuration, master data etc. cleaner and easier, but at the same time the ability to interop with the shell and javascript node environments creates a possibility for the oposite to happen - if you find yourself writing non-trivial shell scripts or javascript code beyond simple expressions in your templates you are probably not using it for the purpose intended, in particular you should not use it for or rely on any side effects caused by the transformation process.

Security considerations

A revibe template is equivalent to code, transforming a template from an untrusted source is equivalent to compiling untrusted source code and running it - don't.

Using recursive revibe (!#!) key prefix implies that the trust you put in your template author is also recursive - the trust boundary for

template.rvb

{
    "!!foo":"file.json"
}

should at least cover the author of template.rvb as file.json will only be JSON.parse()-d

whereas the trust boundary for:

{
    "!#!foo":"file.json"
}

should cover the author of file.json as well becasue the content of file.json itself will be treated as a revibe template.