Datamatic

RxJS + JSON-Schema (Ajv) Based Observable and Validating Data Models and Pipelines

Online Developer Documentation

Goals

• Provide a means to quickly and easily validate complex data-sets
• Look and feel like a standard JS Object for ease of use and adaptability
• Automate data evaluation and transformation

Table of Contents

Installation Instructions

Usage Examples

• Basic Example
• Data Pipelines and Transformation

Developer Guide

• Model Class
  • Model Schemas Config
  • Model Proxy Object
  • model vs $model
• ItemsModel
• PropertiesModel
• BaseModel Class
• Pipeline Class

Installation Instructions

$ npm install datamatic

UMD Usage (React, Angular, Vue et al)

import * as datamatic from "datamatic";

CommonJS Usage for NodeJS

const {Model, Pipeline} = require("datamatic");

DOM Window Usage

    <script src="../../dist/datamatic.window.js"></script>
    <script language="JavaScript">
        const {Model, Pipeline} = window.datamatic;
    </script>

Usage Examples

Basic Example

The example below defines a Model that expects a name value and list of topScores items

const {Model} = require("datamatic");

// JSON-SCHEMA for Scores Collection
const schema = {
    "id": "root#",
    "type": "object",
    "properties": {
        "name": {
            "type": "string",
        },
        "topScores": {
            "type": "array",
            "minItems": 1,
            "maxItems": 3,
            "items": {
                "type": "object",
                "properties": {
                    "name": {
                        "type": "string"
                    },
                    "score": {
                        "type": "integer",
                        "default": 0
                    }
                },
                "required": ["name"]
            }
        }
    },
    required: ["name", "topScores"],
};


// instantiate our Model
const obj = new Model({schemas: [schema]});

// subscribes an observer to the Model
obj.subscribe({
    next: function (ref) {
        console.log("\t>> update succeeded!\n\t%s\n\t%s\n\n",
            "current object state:", "" + JSON.stringify(ref));
    },
    complete: function (ref) {
        console.log("\t>> %s",
            "object is frozen and no longer editable");
    },
    error: function (e) {
        console.log("\t>> update FAILED with error:\n\t%s\n",
            JSON.stringify(e));
        console.log("\tcurrent object state:\n\t%s\n", obj);
    },
});

// populate the Model with data
// -- this will trigger the "next" notification
obj.model = {
    name: "JSONville",
    topScores: [{
        name: "Player 1",
        score: 12300000,
    }, {
        name: "Player 2",
        score: 45600000,
    }, {
        name: "Player 3",
        score: 78900000,
    }]
};

// update the model
// this will trigger the next notification
obj.model.topScores[0].score++;

// invalid attempt update the model
// this will trigger the error notification
// reason: "topScores/items/score" is type is integer 
obj.model.topScores[0].score = "1234";

// invalid attempt update the model
// this will trigger the error notification
// reason: "topScores" is marked as required
delete obj.model.topScores;

Refer to the examples demo in ./examples/basic-usage for more usage examples

Data Pipelines and Transformation

const {Pipeline} = require("datamatic");

/*
    defines a schema that requires name, age and active attributes
    filters out all items that do not conform to JSON-Schema below
 */
const schema = {
    type: "object",
    required: ["name", "age", "active"],
    properties: {
        name: {
            $comment: "names must be in form: First Middle|MI Last",
            type: "string",
            pattern: "^[a-zA-Z]{1,24}\\s?[a-zA-Z]?\\s+[a-zA-Z]{1,24}$",
        },
        age: {
            $comment: "age must be a number equal to or higher than 21 and lower than 100",
            type: "number",
            minimum: 21,
            maximum: 100,
        },
        active: {
            $comment: "active must equal true",
            type: "boolean",
            const: true,
        },
    },
};


const pipeline = new Pipeline(
    [
        // By nesting an item schema within an iterator, the schema is applied as a filter
        schema,
        // the list can go on ...
    ],
    // the list can go on ...
);

pipeline.subscribe({
    // should only contain active people who are 21 and over and name pattern match
    next: (d) => console.log(`\nfiltered results:\n${JSON.stringify(d)}`),
    // it should not encounter an error unless it is critical, so full stop
    error: (e) => console.error(`\ngot error:\n${JSON.stringify(e)}`),
});

pipeline.write([
    {name: "Alice Dodson", age: 30, active: false}, // will be filtered because of active status
    {name: "Jim-Bob", age: 21, active: true}, // will be filtered because of name format
    {name: "Bob Roberts", age: 38, active: true}, // will pass
    {name: "Chris Appleton", age: 19, active: true}, // will be filtered because of age
    {name: "Fred Franks", age: 20, active: true}, // will be filtered because of age
    {name: "Sam Smith", age: 25, active: true}, // will pass
    {name: "", active: null}, // will be filtered because of invalid object format
]);

Developer Guide

Model Class

This class represents the Document entry point

| Method | Arguments | Description | |:--------------|:----------|:-------| | constructor | schemas config (object), [options (object)] | creates new Model instance | | errors [getter] | | retrieves errors (if any) from last json-schema validation | | model [getter/setter] | | retrieves root model proxy object for operation | | getModelsInPath | to (string) | retrieves models at given path | | getSchemaForKey | key (string) | retrieves json-schema with given key as ID | | getSchemaForPath | path (string) | retrieves json-schema for model at given path | | schema [getter] | | retrieves json-schema for root model | | subscribe | observers (object) | Subscribes Observers to the Model Model Root | | subscribeTo | path (string), observers (object) | Subscribes Observers to the Model at path | toString | | retrieves root model as JSON String | | toJSON | | retrieves root model as JSON Object | | validate | path (string), value (object) | validates data at given ath against JSON-Schema | | static fromJSON | json (string | object) | creates new Model from static method |

Model Schemas Config

| Property | Type | Description | |:--------------|:----------|:-------| | meta | array | Array of MetaSchema references to validate Schemas against | schemas | array | Array of Schema references to validate data against | use | string | key/id of schema to use for data validation

Model Proxy Object

This is the Data Model most usage will be based around. It is a Proxy Object that has traps for all operations that alter the state of the given Array or Object

| Property | Type | Description | |:--------------|:----------|:-------| | $model | (PropertiesModel | ItemsModel) | references Proxy Object owner class

model vs $model

In usage, model always references the Proxied Data Model for validation and operation where $model references the owner Model Class

example:

 const _owner = new Model({schemas: [schema]});
 
 // access the root model:
 console.log(`JSON.stringify(_owner.model)`);
 
 // access the model's owner Model Class:
 const owner = _owner.model.$model;
 console.log(`is frozen: ${owner.isFrozen}`);
 
 // call toString on Owner
 console.log(`stringified: ${owner}`);
 
 // obtain model from  it's Owner
 console.log(`stringified: ${JSON.stringify(owner.model)}`);
 

ItemsModel

subclass of Model Class

Represents an Items (Array) entry in the given schema Note: the model param presents a Proxied Array, with all Array.prototype methods trapped and validatable

| Method | Arguments | Description | |:--------------|:----------|:-------| | model [getter/setter] | | setter/getter for model proxy object for operation |

PropertiesModel

subclass of Model Class

Represents an Properties (Object} entry in the given schema

| Method | Arguments | Description | |:--------------|:----------|:-------| | get | key (string) | applies Object.freeze to model hierarchy | | model [getter/setter] | | setter/getter for model proxy object for operation | | set | key (string), value (any) | applies Object.freeze to model hierarchy |

BaseModel Class

| Method | Arguments | Description | |:--------------|:----------|:-------| | freeze | | applies Object.freeze to model hierarchy | | isDirty [getter] | | returns dirtyness of model heirarchy (is dirty if operation in progress) | | isFrozen [getter] | | returns Object.freeze status of Model hierarchy | | jsonPath [getter] | | retrieves json path string for Model instance. eg: "this.is.my.path" | | objectID [getter] | | retrieves Unique ObjectID of Model instance | | options [getter] | | retrieves options passed to Model instance | | path [getter] | | retrieves json-schema path string for Model instance. eg: "#/this/is/my/path" | | parent [getter] | | retrieves Model's parent Model instance | | pipeline | ...(pipes | schemas) | returns Pipeline instance for operating on model | | reset | | resets model to initial state if operation is valid | | root [getter] | | retrieves root Model instance | | model [getter] | | retrieves Model's internal Model Document instance | | subscribe | observers (object) | Subscribes Observers to the Model Model Root | | subscribeTo | path (string), observers (object) | Subscribes Observers to the Model at path | | toString | | retrieves root model as JSON String | | toJSON | | retrieves root model as JSON Object | | validate | path (string), value (object) | validates data at given ath against JSON-Schema | | validationPath [getter] | | retrieves json-schema path string for Model validation |

Pipeline Class

| Method | Arguments | Description | |:--------------|:----------|:-------| | constructor | ...pipesOrSchemas | class constructor method | | errors [getter] | | retrieves errors (if any) from last json-schema validation | | exec | data (object | array | string | number | boolean)| executes pipeline's callback with data without writing to pipeline | | subscribe | handler (object | function | schema | array)| subscribes to pipeline output notifications | | toJSON | | Provides current state of pipeline output as JSON | | toString | | Provides current state of pipeline output as JSON string | | clone | | returns clone of current pipeline segment | | close | | terminates input on pipeline segment | | writable [getter] | | Returns write status of pipeline | | link | target (Pipeline), ...pipesOrSchemas | links pipeline segment to direct output to target pipeline | | merge | ...(pipes | schemas) | merges multiple pipes into single output | | once | | informs pipeline to close after first notification | | pipeline | ...(pipes | schemas) | returns new chained pipeline segment | | sample | nth | Returns product of Nth occurrence of pipeline execution | | split | ...(pipes | schemas) | creates array of new pipeline segments that run in parallel | | tap | | Provides current state of pipeline output. alias for toJSON | | throttle | rate (number) | Limit notifications to rate based on time interval | | unthrottle | discardCacheQueue (boolean) | Clears throttle interval. Optionally discards contents of Pipeline cache | | unlink | target (Pipeline)| unlinks pipeline segment from target pipeline | | write | data (object | array | string | number | boolean)| writes data to pipeline |