data-point

v3.5.0

Published

3 months ago

Data Processing and Transformation Utility

Downloads

317

0High
0Medium
0Low

acatl

raingerber

tim.mahon.d

reduce transform process functional compose aggregate

DataPoint

JavaScript Utility for collecting, processing and transforming data.

DataPoint helps you reason with and streamline your data processing layer. With it you can collect, process, and transform data from multiple sources, and deliver the output in a tailored format to your end consumer.

Prerequisites

Node v8 LTS or higher

Installing

npm install --save data-point

Getting Started

DataPoint provides the following mechanisms for transforming data:

Reducers - these are the simplest transformations; think of them as DataPoint "primitives"
Entities - these are more complex transformations that are defined using one or more reducers
Middleware - middleware functions give the user more control when resolving entities; they're useful to implement caching and other metatasks

The following examples demonstrate some of these concepts. For detailed API documentation, you can jump into the DataPoint.create section and move from there.

Additionally, there is a Hello World YouTube tutorial that explains the basics of DataPoint.

Hello World Example

Trivial example of transforming a given input with a function reducer.

const DataPoint = require('data-point')
// create DataPoint instance
const dataPoint = DataPoint.create()

// function reducer that concatenates
// accumulator.value with 'World'
const reducer = (input) => {
  return input + ' World'
}

// applies reducer to input
dataPoint
  .resolve(reducer, 'Hello')
  .then((output) => {
    // 'Hello World'
    console.log(output) 
  })

Example at: examples/hello-world.js

Fetching remote services

Based on an initial feed, fetch and aggregate results from multiple remote services.

Using the amazing swapi.co service, the example below gets information about a planet and the residents of that planet.

const DataPoint = require('data-point')

// create DataPoint instance
const dataPoint = DataPoint.create()

const {
  Request,
  Model,
  Schema,
  map
} = DataPoint

// schema to verify data input
const PlanetSchema = Schema('PlanetSchema', {
  schema: {
    type: 'object',
    properties: {
      planetId: {
        $id: '/properties/planet',
        type: 'integer'
      }
    }
  }
})

// remote service request
const PlanetRequest = Request('Planet', {
  // {value.planetId} injects the
  // value from the accumulator
  // creates: https://swapi.co/api/planets/1/
  url: 'https://swapi.co/api/planets/{value.planetId}'
})

const ResidentRequest = Request('Resident', {
  // check input is string
  inputType: 'string',
  url: '{value}'
})

// model entity to resolve a Planet
const ResidentModel = Model('Resident', {
  inputType: 'string',
  value: [
    // hit request:Resident
    ResidentRequest,
    // extract data
    {
      name: '$name',
      gender: '$gender',
      birthYear: '$birth_year'
    }
  ]
})

// model entity to resolve a Planet
const PlanetModel = Model('Planet', {
  inputType: PlanetSchema,
  value: [
    // hit request:Planet data source
    PlanetRequest,
    // map result to an object reducer
    {
      // map name key
      name: '$name',
      population: '$population',
      // residents is an array of urls
      // eg. https://swapi.co/api/people/1/
      // where each url gets mapped
      // to a model:Resident
      residents: ['$residents', map(ResidentModel)]
    }
  ]
})

const input = {
  planetId: 1
}

dataPoint.resolve(PlanetModel, input)
  .then((output) => {
    console.log(output)
    /*
    output -> 
    { 
      name: 'Tatooine',
      population: 200000,
      residents:
      [ 
        { name: 'Luke Skywalker', gender: 'male', birthYear: '19BBY' },
        { name: 'C-3PO', gender: 'n/a', birthYear: '112BBY' },
        { name: 'Darth Vader', gender: 'male', birthYear: '41.9BBY' },
        ...
      ] 
    }
    */
  })

Example at: examples/full-example-instances.js

API

create

Static method that creates a DataPoint instance.

SYNOPSIS

DataPoint.create([options])

Arguments

| Argument | Type | Description | |:---|:---|:---| | options | Object (optional) | This parameter is optional, as are its properties (values, entities, and entityTypes). You may configure the instance later through the instance's API. |

The following table describes the properties of the options argument.

| Property | Type | Description | |:---|:---|:---| | values | Object | Hash with values you want exposed to every Reducer | | entities | Object | Application's defined entities | | entityTypes | Object | Custom Entity Types |

RETURNS

DataPoint instance.

DataPoint.create example

const DataPoint = require('data-point')
const dataPoint = DataPoint.create()

Create the DataPoint object and set options argument:

const DataPoint = require('data-point')
const dataPoint = DataPoint
  .create({
    values: {
      foo: 'bar'
    },
    entities: {
      'reducer:HelloWorld': (input) => {
        return `hello ${input}!!`
      }
    }
  })

createReducer

Static method that creates a reducer, which can be executed with resolve or transform.

SYNOPSIS

DataPoint.createReducer(source:*, [options:Object]):Reducer

ARGUMENTS

| Argument | Type | Description | |:---|:---|:---| | source | * | Spec for any of the supported Reducer types | | options | Object | Optional config object |

Options

| Property | Type | Description | |:---|:---|:---| | default | * | Default value for the reducer. Setting this value is equivalent to using the withDefault reducer helper. |

resolve

Execute a Reducer against an input value. This function supports currying and will be executed when at least the first 2 parameters are provided.

SYNOPSIS

dataPoint.resolve(reducer:Reducer, input:*, options:Object):Promise(output:*)

This method returns a Promise with the final output value.

ARGUMENTS

EXAMPLES:

Hello World example.
With options example.

transform

This method is similar to dataPoint.resolve. The differences between the methods are:

.transform() accepts an optional third parameter for node style callback.
.transform() returns a Promise that resolves to the full Accumulator object instead of accumulator.value. This may come in handy if you want to inspect other values from the transformation.

SYNOPSIS

// as promise
dataPoint.transform(reducer:Reducer, input:*, options:Object):Promise(acc:*)
// as nodejs callback function
dataPoint.transform(reducer:Reducer, input:*, options:Object, done:Function)

This method will return a Promise if done is omitted.

ARGUMENTS

| Argument | Type | Description | |:---|:---|:---| | reducer | Reducer | Reducer that manipulates the input. | | input | * | Input value that you want to transform. If none, pass null or empty object {}. | | options | Object | Options within the scope of the current transformation | | done | function (optional) | Error-first Node.js style callback with the arguments (error, accumulator). The second parameter is an Accumulator object where accumulator.value is the actual result of the transformation.

transform options

The following table describes the properties of the options argument.

| Property | Type | Description | |:---|:---|:---| | locals | Object | Hash with values you want exposed to every reducer. See example. | | trace | boolean | Set this to true to trace the entities and the time each one is taking to execute. Use this option for debugging. |

resolveFromAccumulator

Execute a Reducer from a provided Accumulator. This function will attempt at resolving a reducer providing an already constructed Accumulator Object. It will take the value provided in the Accumulator object to use as the input.

SYNOPSIS

dataPoint.resolveFromAccumulator(reducer:Reducer, acc:Accumulator):Promise(acc:Accumulator)

This method returns a Promise with the final output value.

ARGUMENTS

| Argument | Type | Description | |:---|:---|:---| | reducer | Reducer | Reducer that manipulates the input. | | acc | Accumulator | Reducer's accumulator Object. The main property is value, which is the value the reducer will use as its input. |

addEntities

This method adds new entities to a DataPoint instance.

SYNOPSIS

When defining new entities, <EntityType> must refer to either a built-in type like 'model' or a custom entity type. <EntityId> should be unique for each type; for example, model:foo and hash:foo can both use the foo ID, but an error is thrown if model:foo is defined twice.

dataPoint.addEntities({
  '<EntityType>:<EntityId>': { ... },
  '<EntityType>:<EntityId>': { ... },
  ...
})

OPTIONS

| Part | Type | Description | |:---|:---|:---| | EntityType | string | valid entity type | | EntityId | string | unique entity ID |

addValue

Stores any value to be accessible via Accumulator.values. This object can also be set by passing a values property to DataPoint.create.

SYNOPSIS

dataPoint.addValue(objectPath, value)

ARGUMENTS

| Argument | Type | Description | |:---|:---|:---| | objectPath | string | object path where you want to add the new value. Uses _.set to append to the values object | | value | * | anything you want to store |

Accumulator

This object is passed to reducers and middleware callbacks; it has contextual information about the current transformation or middleware that's being resolved.

The accumulator.value property is the current input data. This property should be treated as a read-only immutable object. This helps ensure that your reducers are pure functions that produce no side effects. If the value is an object, use it as your initial source for creating a new object.

Properties exposed:

| Key | Type | Description | |:---|:---|:---| | value | Object | Value to be transformed. | | initialValue | Object | Initial value passed to the entity. You can use this value as a reference to the initial value passed to your Entity before any reducer was applied. | | values | Object | Access to the values stored via dataPoint.addValue. | | params | Object | Value of the current Entity's params property. (for all entities except Reducer) | | locals | Object | Value passed from the options argument when executing dataPoint.transform. | | reducer | Object | Information relative to the current Reducer being executed. | | debug | Function | debug method with scope data-point |

Reducers

Reducers are used to transform values asynchronously. DataPoint supports the following reducer types:

Path Reducer

A path reducer is a string that extracts a path from the current Accumulator value (which must be an Object). It uses lodash's _.get behind the scenes.

SYNOPSIS

'$[.|..|<path>]'

OPTIONS

| Option | Description | |:---|:---| | $ | Reference to current accumulator.value. | | $.. | Gives full access to accumulator properties (i.e. $..params.myParam). | | $path | Object path notation to extract data from accumulator.value. | | $path[] | Appending [] will map the reducer to each element of an input array. If the current accumulator value is not an array, the reducer will return undefined.

Root path $

EXAMPLES:

const DataPoint = require('data-point')
const dataPoint = DataPoint.create()

const input = {
  a: {
    b: [
      'Hello World'
    ]
  }
}

dataPoint
  .resolve('$', input)
  .then((output) => {
    assert.strictEqual(output, input)
  })

Access accumulator reference

const DataPoint = require('data-point')
const dataPoint = DataPoint.create()

const input = {
  a: {
    b: [
      'Hello World'
    ]
  }
}

dataPoint
  .resolve('$..value', input)
  .then(output => {
    assert.strictEqual(output input)
  })

Object Path

const DataPoint = require('data-point')
const dataPoint = DataPoint.create()

const input = {
  a: {
    b: [
      'Hello World'
    ]
  }
}

dataPoint
  .resolve('$a.b[0]', input)
  .then(output => {
    assert.strictEqual(output, 'Hello World')
  })

Example at: examples/reducer-path.js

Object Map

const DataPoint = require('data-point')
const dataPoint = DataPoint.create()

const input = [
  {
    a: {
      b: 'Hello World'
    }
  },
  {
    a: {
      b: 'Hello Solar System'
    }
  },
  {
    a: {
      b: 'Hello Universe'
    }
  }
]

dataPoint
  .resolve('$a.b[]', input)
  .then(output => {
    assert.deepStrictEqual(output, ['Hello World', 'Hello Solar System', 'Hello Universe'])
  })

Function Reducer

A function reducer allows you to use a function to apply a transformation. There are several ways to define a function reducer:

Synchronous function that returns new value
Asynchronous function that returns a Promise
Asynchronous function with callback parameter
Asynchronous function through async/await (only if your environment supports it)

IMPORTANT: Be careful with the parameters passed to your function reducer; DataPoint relies on the number of arguments to detect the type of function reducer it should expect.

Returning a value

The returned value is used as the new value of the transformation.

SYNOPSIS

const name = (input:*, acc:Accumulator) => {
  return newValue
}

Reducer's arguments

| Argument | Type | Description | |:---|:---|:---| | input | * | Reference to acc.value | | acc | Accumulator | Current reducer's accumulator Object. The main property is value, which is the current reducer's value. |

 const reducer = (input, acc) => {
  return input + ' World'
}

dataPoint
  .resolve(reducer, 'Hello')
  .then((output) => {
    assert.strictEqual(output, 'Hello World')
  })

Example at: examples/reducer-function-sync.js

Returning a Promise

If you return a Promise its resolution will be used as the new value of the transformation. Use this pattern to resolve asynchronous logic inside your reducer.

SYNOPSIS

const name = (input:*, acc:Accumulator) => {
  return Promise.resolve(newValue)
}

Reducer's arguments

const reducer = (input, acc) => {
  // input is a reference to acc.value
  return Promise.resolve(acc.value + ' World')
}

dataPoint
  .resolve(reducer, 'Hello')
  .then((output) => {
    assert.strictEqual(output, 'Hello World')
  })

Example at: examples/reducer-function-promise.js

With a callback parameter

Accepting a third parameter as a callback allows you to execute an asynchronous block of code. This should be an error-first, Node.js style callback with the arguments (error, value), where value will be the value passed to the next transform; this value becomes the new value of the transformation.

SYNOPSIS

const name = (input:*, acc:Accumulator, next:function) => {
  next(error:Error, newValue:*)
}

Reducer's arguments

| Argument | Type | Description | |:---|:---|:---| | input | * | Reference to acc.value | | acc | Accumulator | Current reducer's accumulator Object. The main property is acc.value, which is the current reducer's value. | | next | Function(error,value) | Node.js style callback, where value is the value to be passed to the next reducer.

const reducer = (input, acc, next) => {
  next(null, input + ' World')
}

dataPoint
  .resolve(reducer, 'Hello')
  .then((output) => {
    assert.strictEqual(output, 'Hello World')
  })

Example at: examples/reducer-function-with-callback.js

const throwError = (error, acc, next) => {
  // passing first argument will be
  // handled as an error by the transform
  next(new Error('oh noes!!'))
}

dataPoint
  .transform(throwError, 'Hello')
  .catch((error) => {
    console.assert(error instanceof Error)
    console.log(error.toString()) // 'Error: oh noes!!'
  })

Example at: examples/reducer-function-error.js

Object Reducer

These are plain objects where the value of each key is a Reducer. They're used to aggregate data or transform objects. You can add constants with the constant reducer helper, which is more performant than using a function reducer:

const { constant } = require('data-point')

const objectReducer = {
  // in this case, x and y both resolve to 42, but DataPoint
  // can optimize the resolution of the constant value
  x: () => 42,
  y: constant(42)
}

const inputData = {
  x: {
    y: {
      z: 2
    }
  }
}

const objectReducer = {
  y: '$x.y',
  zPlusOne: ['$x.y.z', (input) => input + 1]
}

// output from dataPoint.transform(objectReducer, inputData):

{
  y: {
    z: 2
  },
  zPlusOne: 3 
}

const dataPoint = require('data-point').create()

dataPoint.addEntities({
  'request:Planet': {
    url: 'https://swapi.co/api/planets/{value}'
  }
})

const objectReducer = {
  tatooine: ['$tatooine', 'request:Planet'],
  alderaan: ['$alderaan', 'request:Planet']
}

const planetIds = {
  tatooine: 1,
  alderaan: 2
}

dataPoint.resolve(objectReducer, planetIds)
  .then(output => {
    // do something with the aggregated planet data!
  })

Each of the reducers, including the nested ones, are resolved against the same accumulator value. This means that input objects can be rearranged at any level:

const inputData = {
  a: 'A',
  b: 'B',
  c: {
    x: 'X',
    y: 'Y'
  }
})

// some data will move to a higher level of nesting,
// but other data will move deeper into the object
const objectReducer = {
  x: '$c.x',
  y: '$c.y',
  z: {
    a: '$a',
    b: '$b'
  }
}

// output from dataPoint.resolve(objectReducer, inputData):

{
  x: 'X',
  y: 'Y',
  z: {
    a: 'A',
    b: 'B'
  }
}

Each of the reducers might contain more object reducers (which might contain other reducers, and so on). Notice how the output changes based on the position of the object reducers in the two expressions:

const inputData = {
  a: {
    a: 1,
    b: 2
  }
}

const objectReducer = {
  x: [
    '$a',
    // this comes second, so it's resolved
    // against the output from the '$a' transform
    {
      a: '$a'
    }
  ],
  y: [
    // this comes first, so it's resolved
    // against the main input to objectReducer
    {
      a: '$a'
    },
    '$a'
  ]
}

// output from dataPoint.resolve(objectReducer, inputData):

{
  x: {
    a: 1
  },
  y: {
    a: 1,
    b: 2
  }
}

An empty object reducer will resolve to an empty object:

const reducer = {}

const input = { a: 1 }

dataPoint.resolve(reducer, input) // => {}

Entity Reducer

An entity instance reducer is used to apply a given entity with to the current Accumulator.

See the Entities section for information about the supported entity types.

OPTIONS

| Option | Type | Description | |:---|:---|:---| | ? | String | Only execute entity if acc.value is not equal to false, null or undefined. | | EntityType | String | Valid Entity type. | | EntityID | String | Valid Entity ID. Appending [] will map the reducer to each element of an input array. If the current accumulator value is not an array, the reducer will return undefined. |

const {
  Model,
  Request
} = require('data-point')

const PersonRequest = Request('PersonRequest', {
  url: 'https://swapi.co/api/people/{value}'
})

const PersonModel = Model('PersonModel', {
  value: {
    name: '$name',
    birthYear: '$birth_year'
  }
})

dataPoint
  .resolve([PersonRequest, PersonModel], 1)
  .then((output) => {
    assert.deepStrictEqual(output, {
      name: 'Luke Skywalker',
      birthYear: '19BBY'
    })
  })

Example at: examples/reducer-entity-instance.js

Entity By Id Reducer

An entity reducer is used to execute an entity with the current Accumulator as the input.

Appending [] to an entity reducer will map the given entity to each element of an input array. If the current accumulator value is not an array, the reducer will return undefined.

See the Entities section for information about the supported entity types.

SYNOPSIS

'[?]<EntityType>:<EntityId>[[]]'

OPTIONS

const input = {
  a: {
    b: 'Hello World'
  }
}

const toUpperCase = (input) => {
  return input.toUpperCase()
}

dataPoint.addEntities({
  'reducer:getGreeting': '$a.b',
  'reducer:toUpperCase': toUpperCase,
})

// resolve `reducer:getGreeting`,
// pipe value to `reducer:toUpperCase`
dataPoint
  .resolve(['reducer:getGreeting | reducer:toUpperCase'], input)
  .then((output) => {
    assert.strictEqual(output, 'HELLO WORLD')
  })

const input = {
  a: [
    'Hello World',
    'Hello Laia',
    'Hello Darek',
    'Hello Italy',
  ]
}

const toUpperCase = (input) => {
  return input.toUpperCase()
}

dataPoint.addEntities({
  'reducer:toUpperCase': toUpperCase
})

dataPoint
  .resolve(['$a | reducer:toUpperCase[]'], input)
  .then((output) => {
    assert.strictEqual(output[0], 'HELLO WORLD')
    assert.strictEqual(output[1], 'HELLO LAIA')
    assert.strictEqual(output[2], 'HELLO DAREK')
    assert.strictEqual(output[3], 'HELLO ITALY')
  })

List Reducer

A list reducer is an array of reducers where the result of each reducer becomes the input to the next reducer. The reducers are executed serially and asynchronously. It's possible for a list reducer to contain other list reducers.

| List Reducer | Description | |:---|:---| | ['$a.b', (input) => { ... }] | Get path a.b, pipe value to function reducer | | ['$a.b', (input) => { ... }, 'hash:Foo'] | Get path a.b, pipe value to function reducer, pipe result to hash:Foo |

IMPORTANT: an empty list reducer will resolve to undefined. This mirrors the behavior of empty functions.

const reducer = []

const input = 'INPUT'

dataPoint.resolve(reducer, input) // => undefined

Conditionally execute an entity

Only execute an entity if the accumulator value is not equal to false, null or undefined. If the conditional is not met, the entity will not be executed and the value will remain the same.

const people = [
  {
    name: 'Luke Skywalker',
    swapiId: '1'
  },
  {
    name: 'Yoda',
    swapiId: null
  }
]

dataPoint.addEntities({
  'request:getPerson': {
    url: 'https://swapi.co/api/people/{value}'
  },
  'reducer:getPerson': {
    name: '$name',
    // request:getPerson will only
    // be executed if swapiId is
    // not false, null or undefined
    birthYear: '$swapiId | ?request:getPerson | $birth_year'
  }
})

dataPoint
  .resolve('reducer:getPerson[]', people)
  .then((output) => {
    assert.deepStrictEqual(output, [
      {
        name: 'Luke Skywalker',
        birthYear: '19BBY'
      },
      {
        name: 'Yoda',
        birthYear: undefined
      }
    ])
  })

Example at: examples/reducer-conditional-operator.js

Reducer Helpers

Reducer helpers are factory functions for creating reducers. They're accessed through the DataPoint Object:

const {
  assign,
  constant,
  filter,
  find,
  map,
  parallel,
  withDefault
} = require('data-point')

assign

The assign reducer creates a new Object by resolving the provided Reducer and merging the result with the current accumulator value. It uses Object.assign internally.

SYNOPSIS

assign(reducer:Reducer):Object

Reducer's arguments

| Argument | Type | Description | |:---|:---|:---| | reducer | Reducer | Result from this reducer will be merged into the current accumulator.value. By convention, this reducer should return an Object. |

EXAMPLE:

const input = {
  a: 1
}

// merges the object reducer with
// accumulator.value
const reducer = DataPoint.assign({
  c: '$b.c'
})

dataPoint
  .resolve(reducer, input)
  .then(output => {
    /*
     output --> {
      a: 1,
      b: {
        c: 2
      },
      c: 2
    }
    */
  })

Example at: examples/reducer-helper-assign.js

map

The map reducer creates a new array with the results of applying the provided Reducer to every element in the input array.

SYNOPSIS

map(reducer:Reducer):Array

Reducer's arguments

| Argument | Type | Description | |:---|:---|:---| | reducer | Reducer | The reducer will get applied to each element in the array. |

EXAMPLE:

const input = [{
  a: 1
}, {
  a: 2
}]

// get path `a` then multiply by 2
const reducer = DataPoint.map(
  ['$a', (input) => input * 2]
)

dataPoint
  .resolve(reducer, input)
  .then(output => {
    // output -> [2, 4]
  })

Example at: examples/reducer-helper-map.js

filter

The filter reducer creates a new array with elements that resolve as truthy when passed to the given Reducer.

SYNOPSIS

filter(reducer:Reducer):Array

Reducer's arguments

| Argument | Type | Description | |:---|:---|:---| | reducer | Reducer | Reducer result is used to test for truthy on each element of the array. |

EXAMPLE:

const input = [{ a: 1 }, { a: 2 }]

// filters array elements that are not
// truthy for the given list reducer
const reducer = DataPoint.filter(
  ['$a', (input) => input > 1]
)

dataPoint
  .resolve(reducer, input) 
  .then(output => {
    // output ->  [{ a: 2 }]
  })

Example at: examples/reducer-helper-filter.js

find

The find reducer returns the first element of an array that resolves to truthy when passed through the provided Reducer. It returns undefined if no match is found.

SYNOPSIS

find(reducer:Reducer):*

Reducer's arguments

| Argument | Type | Description | |:---|:---|:---| | reducer | Reducer | Reducer result is used to test for truthy on each element of the array. |

EXAMPLE:

const input = [{ a: 1 }, { b: 2 }]

// the $b reducer is truthy for the
// second element in the array
const reducer = DataPoint.find('$b')

dataPoint
  .resolve(reducer, input) 
  .then(output => {
    // output -> { b: 2 }
  })

Example at: examples/reducer-helper-find.js

constant

The constant reducer always returns the given value. If a reducer is passed it will not be evaluated. This is primarily meant to be used in object reducers.

SYNOPSIS

constant(value:*):*

Reducer's arguments

| Argument | Type | Description | |:---|:---|:---| | value | * | The value the reducer should return |

EXAMPLE:

const input = {
  a: 1,
  b: 2
}

const reducer = {
  a: '$a',
  b: DataPoint.constant({
    a: '$a',
    b: 3
  })
}

dataPoint
  .resolve(reducer, input) 
  .then(output => {
    // {
    //   a: 1,
    //   b: {
    //     a: '$a',
    //     b: 3
    //   }
    // }
    }
  })

const input = {
  b: 1
}

// object reducer that contains a path reducer ('$a')
let reducer = {
  a: '$b'
}

dataPoint.resolve(reducer, input) // => { a: 1 }

// both the object and the path will be treated as
// constants instead of being used to create reducers
reducer = DataPoint.constant({
  a: '$b'
})

dataPoint.resolve(reducer, input) // => { a: '$b' }

parallel

This resolves an array of reducers. The output is a new array where each element is the output of a reducer; this contrasts with list reducers, which return the output from the last reducer in the array.

SYNOPSIS

parallel(reducers:Array<Reducer>):Array

Reducer's arguments

| Argument | Type | Description | |:---|:---|:---| | reducers | Array | Source data to create an array of reducers |

EXAMPLE:

const reducer = DataPoint.parallel([
  '$a',
  ['$b', (input) => input + 2] // list reducer
])

const input = {
  a: 1,
  b: 2
}

dataPoint.resolve(reducer, input) // => [1, 4]

withDefault

The withDefault reducer adds a default value to any reducer type. If the reducer resolves to null, undefined, NaN, or '', the default is returned instead.

SYNOPSIS

withDefault(source:*, value:*):*

Reducer's arguments

| Argument | Type | Description | |:---|:---|:---| | source | * | Source data for creating a Reducer | | value | * | The default value to use (or a function that returns the default value) |

The default value is not cloned before it's returned, so it's good practice to wrap any Objects in a function.

EXAMPLE:

const input = {
  a: undefined
}

// adds a default to a path reducer
const r1 = DataPoint.withDefault('$a', 50)

dataPoint.resolve(r1, input) // => 50

// passing a function is useful when the default value is
// an object, because it returns a new object every time
const r2 = withDefault('$a', () => {
  return { b: 1 }
})

dataPoint.resolve(r2, input) // => { b: 1 }

Entities

Entities are used to transform data by composing multiple reducers, they can be created as non-registered or registered entities.

Instance entities - are entity objects created directly with an Entity Factory, they are meant to be used as a entity reducer.
Registered entities - are entity objects which are registered and cached in a DataPoint instance, they are meant to be used as a registered entity reducer.

Registered entities may be added to DataPoint in two different ways:

With the DataPoint.create method (as explained in the setup examples)
With the dataPoint.addEntities instance method

See built-in entities for information on what each entity does.

Instance Entity

Entities can be created from these factory functions:

const {
  Entry,
  Model,
  Reducer,
  Collection,
  Hash,
  Request,
  Control,
  Schema
} = require('data-point')

SYNOPSIS

Each factory has the following signature:

Factory(name:String, spec:Object):EntityInstance

ARGUMENTS

| Argument | Type | Description | |:---|:---|:---| | name | string | The name of the entity; this will be used to generate an entity ID with the format <entityType>:<name> | | spec | Object | The source for generating the entity |

Example

const DataPoint = require('data-point')
const { Model } = DataPoint

const dataPoint = DataPoint.create()

const HelloWorld = Model('HelloWorld', {
  value: input => ({
    hello: 'world'
  })
})

// to reference it we use the actual entity instance
dataPoint.resolve(HelloWorld, {})
  .then(value => {
    console.assert(value, {
      hello: 'world'
    })
  })

Registered Entity

You may register an entity through DataPoint.create or dataPoint.addEntities.

Example

const DataPoint = require('data-point')

dataPoint = DataPoint.create({
  entities: {
    'model:HelloWorld', {
      value: input => ({
        hello: 'world'
      } 
    }
  }
})

// to reference it we use a string with its registered id
dataPoint.resolve('model:HelloWorld', {})
  .then(value => {
    console.assert(value, {
      hello: 'world'
    })
  })

Entity Base API

All entities share a common API (except for Reducer).

{
  // type checks the entity's input
  inputType: String | Reducer,

  // executes --before-- any modifier
  before: Reducer,
  
  // executes --after-- any modifier
  after: Reducer,
  
  // type checks the entity's output
  outputType: String | Reducer,

  // executes in case there is an error at any
  // point of the entire transformation
  error: Reducer,
  
  // this object allows you to store and eventually
  // access it at any given time on any reducer
  params: Object
}

Properties exposed:

| Key | Type | Description | |:---|:---|:---| | inputType | String, Reducer | type checks the entity's input value, but does not mutate it | | before | Reducer | reducer to be resolved before the entity resolution | | after | Reducer | reducer to be resolved after the entity resolution | | outputType | String, Reducer | type checks the entity's output value, but does not mutate it | | error | Reducer | reducer to be resolved in case of an error (including errors thrown from the inputType and outputType reducers) | | params | Object | user defined Hash that will be passed to every transform within the context of the transform's execution |

Entity type check

You can use inputType and outputType for type checking an entity's input and output values. Type checking does not mutate the result.

Built in type checks:

To use built-in type checks, you may set the value of inputType or outputType to: 'string', 'number', 'boolean', 'function', 'error', 'array', or 'object'.

This example uses a Model Entity, for information on what a model is please go to the Model Entity section.

const dataPoint = DataPoint.create()

dataPoint.addEntities({
  'model:getName': {
    value: '$name',
    outputType: 'string'
  }
})

const input = {
  name: 'DataPoint'
}

dataPoint.resolve('model:getName', input)
  .then(output => {
    // output -> 'DataPoint'
  })

Custom type checking:

You may also type check with a Schema Entity, or by creating a Reducer with the createTypeCheckReducer function.

SYNOPSIS

DataPoint.createTypeCheckReducer(typeCheckFunction, [expectedType])

ARGUMENTS

| Argument | Type | Description | |:---|:---|:---| | typeCheckFunction | Function<Boolean|String> | Return true when the input is valid; otherwise, an error will be thrown. If the function returns a string, that will be appended to the error message. | | expectedType | string (optional) | The expected type; this will also be used in the error message. |

  const DataPoint = require('data-point')

  const { createTypeCheckReducer } = DataPoint

  const isNonEmptyArray = input => Array.isArray(input) && input.length > 0

  const dataPoint = DataPoint.create({
    entities: {
      'model:get-first-item': {
        inputType: createTypeCheckReducer(isNonEmptyArray, 'non-empty-array'),
        value: input => input[0]
      }
    }
  })

In this example we are using a Schema Entity to check the inputType.

const dataPoint = DataPoint.create()

dataPoint.addEntities({
  'model:getName': {
    // assume schema:RepoSchema 
    // exists and checks of the
    // existence of name
    inputType: 'schema:RepoSchema',
    value: '$name'
  }
})

const input = {
  name: 'DataPoint'
}

dataPoint.resolve('model:getName', input)
  .then(output => {
    // output -> DataPoint
  })

Entity Types

DataPoint comes with the following built-in entity types:

Reducer

A Reducer entity is a 'snippet' that you can re-use in other entities. It does not expose the before/after/error/params API that other entities have.

IMPORTANT: Reducer Entities do not support extension.

SYNOPSIS

dataPoint.addEntities({
  'reducer:<entityId>': Reducer
})

For backwards compatibility, the keyword transform can be used in place of reducer:

dataPoint.addEntities({
  'reducer:<entityId>': Reducer
})

const input = {
  a: {
    b: {
      c: [1, 2, 3]
    }
  }
}

const getMax = (input) => {
  return Math.max.apply(null, input)
}

const multiplyBy = (number) => (input) => {
  return input * number
}

dataPoint.addEntities({
  'reducer:foo': ['$a.b.c', getMax, multiplyBy(10)]
})

dataPoint
  .resolve('reducer:foo', input)
  .then((output) => {
    assert.strictEqual(output, 30)
  })

Model

A Model entity is a generic entity that provides the base methods.

SYNOPSIS

dataPoint.addEntities({
  'model:<entityId>': {
    inputType: String | Reducer,
    before: Reducer,
    value: Reducer,
    after: Reducer,
    outputType: String | Reducer,
    error: Reducer,
    params: Object
  }
})

Properties exposed:

| Key | Type | Description | |:---|:---|:---| | inputType | String, Reducer | type checks the entity's input value, but does not mutate it | | before | Reducer | reducer to be resolved before the entity resolution | | after | Reducer | reducer to be resolved after the entity resolution | | outputType | String, Reducer | type checks the entity's output value, but does not mutate it | | error | Reducer | reducer to be resolved in case of an error | | params | Object | user defined Hash that will be passed to every transform within the context of the transform's execution |

Model.value

const input = {
  a: {
    b: {1
      c: [1, 2, 3]
    }
  }
}

const getMax = (input) => {
  return Math.max.apply(null, input)
}

const multiplyBy = (number) => (input) => {
  return input * number
}

dataPoint.addEntities({
  'model:foo': {
    value: ['$a.b.c', getMax, multiplyBy(10)]
  }
})

dataPoint
  .resolve('model:foo', input)
  .then((output) => {
    assert.strictEqual(output, 30)
  })

Example at: examples/entity-model-basic.js

Model.before

const toArray = (input) => {
  return Array.isArray(input)
    ? input
    : [input]
}

dataPoint.addEntities({
  'model:foo': {
    before: toArray,
    value: '$'
  }
})

dataPoint
  .resolve('model:foo', 100)
  .then((output) => {
    assert.deepStrictEqual(output, [100])
  })

Example at: examples/entity-model-before.js

Model.after

const toArray = (input) => {
  return Array.isArray(input)
    ? input
    : [input]
}

dataPoint.addEntities({
  'model:foo': {
    value: '$a.b',
    after: isArray()
  }
})

const input = {
  a: {
    b: [3, 15]
  }
}

dataPoint
  .resolve('model:foo', input)
  .then((output) => {
    assert.deepStrictEqual(output, [3, 15])
  })

Model.error

Any error that happens within the scope of the Entity can be handled by the error transform. To respect the API, error reducers have the same API.

Error handling

Passing a value as the second argument will stop the propagation of the error.

EXAMPLES:

Let's resolve to a non-array value and see how it would be handled, this example will use outputType for type checking.


dataPoint.addEntities({
  'model:getArray': {
    // points to a NON Array value
    value: '$a.b',
    outputType: 'isArray',
    error: (error) => {
      // prints out the error
      // message generated by
      // isArray type check
      console.log(error.message)

      console.log('Value is invalid, resolving to empty array')

      // passing a value will stop
      // the propagation of the error
      return []
    }
  }
})

const input = {
  a: {
    b: 'foo'
  }
}

dataPoint
  .resolve('model:getArray', input)
  .then((output) => {
    assert.deepStrictEqual(output, [])
  })

Example at: examples/entity-model-error-handled.js

const logError = (error) => {
  console.log(error.toString())
  throw error
}

dataPoint.addEntities({
  'model:getArray': {
    value: '$a',
    outputType: 'isArray',
    error: logError
  }
})

const input = {
  a: {
    b: 'foo'
  }
}

dataPoint
  .resolve('model:getArray', input)
  .catch((error) => {
    console.log(error.toString())
  })

Example at: examples/entity-model-error-rethrow.js

Model.params

The params object is used to pass custom data to your entity. This Object is exposed as a property of the Accumulator Object. Which can be accessed via a function reducer, as well as through a path reducer expression.

const multiplyValue = (input, acc) => {
  return input * acc.params.multiplier
}

dataPoint.addEntities({
  'model:multiply': {
    value: multiplyValue,
    params: {
      multiplier: 100
    }
  }
})

dataPoint
  .resolve('model:multiply', 200)
  .then((output) => {
    assert.deepStrictEqual(output, 20000)
  })

dataPoint.addEntities({
  'model:getParam': {
    value: '$..params.multiplier',
    params: {
      multiplier: 100
    }
  }
})

dataPoint.resolve('model:getParam')
  .then((output) => {
    assert.deepStrictEqual(output, 100)
  })

Entry

This entity is very similar to the Model entity. Its main difference is that this entity will default to an empty object { } as its initial value if none was passed. As a best practice, use it as your starting point, and use it to call more complex entities.

SYNOPSIS

dataPoint.addEntities({
  'entry:<entityId>': {
    inputType: String | Reducer,
    before: Reducer,
    value: Reducer,
    after: Reducer,
    outputType: String | Reducer,
    error: Reducer,
    params: Object
  }
})

Properties exposed:

| Key | Type | Description | |:---|:---|:---| | inputType | String, Reducer | type checks the entity's input value, but does not mutate it | | before | Reducer | reducer to be resolved before the entity resolution | | after | Reducer | reducer to be resolved after the entity resolution | | outputType | String, Reducer | type checks the entity's output value, but does not mutate it | | error | Reducer | reducer to be resolved in case of an error | | params | Object | user defined Hash that will be passed to every transform within the context of the transform's execution |

Request

Requests a remote source, using request-promise behind the scenes. The features supported by request-promise are exposed/supported by Request entity.

SYNOPSIS

dataPoint.addEntities({
  'request:<entityId>': {
    inputType: String | Reducer,
    before: Reducer,
    value: Reducer,
    url: StringTemplate,
    options: Reducer,
    after: Reducer,
    outputType: String | Reducer,
    error: Reducer,
    params: Object
  }
})

Properties exposed:

| Key | Type | Description | |:---|:---|:---| | inputType | String, Reducer | type checks the entity's input value, but does not mutate it | | before | Reducer | reducer to be resolved before the entity resolution | | value | Reducer | the result of this reducer is the input when resolving url and options | url | StringTemplate | String value to resolve the request's url | | options | Reducer | reducer that returns an object to use as request-promise options | after | Reducer | reducer to be resolved after the entity resolution | | error | Reducer | reducer to be resolved in case of an error | | outputType | String, Reducer | type checks the entity's output value, but does not mutate it | | params | Object | user defined Hash that will be passed to every reducer within the context of the transform function's execution |

Request.url

Sets the url to be requested.

NOTE: When Request.url is not defined it will use the current Accumulator.value (if the value is of type string) as the Request's url.

dataPoint.addEntities({
  'request:getLuke': {
    url: 'https://swapi.co/api/people/1/'
  }
})

dataPoint.resolve('request:getLuke', {})
  .then(output => {
    /*
    output -> 
    {
      name: 'Luke Skywalker',
      height: '172',
      ...
    }
    */
  })

Example at: examples/entity-request-basic.js

Request.url as StringTemplate

StringTemplate is a string that supports a minimal templating system. You may inject any value into the string by enclosing it within {ObjectPath} curly braces. The context of the string is the Request's Accumulator Object, meaning you have access to any property within it.

Using acc.value property to make the url dynamic:

dataPoint.addEntities({
  'request:getLuke': {
    // inject the acc.value.personId
    url: 'https://swapi.co/api/people/{value.personId}/'
  }
})

const input = {
  personId: 1
}

dataPoint.resolve('request:getLuke', input)
  .then(output => {
    /*
    output -> 
    {
      name: 'Luke Skywalker',
      height: '172',
      ...
    }
    */
  })

Example at: examples/entity-request-string-template.js

Using acc.locals property to make the url dynamic:

dataPoint.addEntities({
  'request:getLuke': {
    url: 'https://swapi.co/api/people/{locals.personId}/'
  }
})

const options = {
  locals: {
    personId: 1
  }
}

dataPoint.resolve('request:getLuke', {}, options)
  .then(output => {
  /*
  output -> 
  {
    name: 'Luke Skywalker',
    height: '172',
    ...
  }
  */
  })

Example at: examples/entity-request-options-locals.js

For more information on acc.locals: Transform Options and Accumulator Objects.

Using constants in the options reducer:

const DataPoint = require('data-point')
const dataPoint = DataPoint.create()

dataPoint.addEntities({
  'request:searchPeople': {
    url: 'https://swapi.co/api/people',
    // options is a Reducer, but values
    // at any level can be wrapped as
    // constants (or just wrap the whole
    // object if all the values are static)
    options: {
      'content-type': DataPoint.constant('application/json')
      qs: {
        // get path `searchTerm` from input
        // to dataPoint.resolve
        search: '$searchTerm'
      }
    }
  }
})

const input = {
  searchTerm: 'r2'
}

// the second parameter to transform is the input value
dataPoint
  .resolve('request:searchPeople', input)
  .then(output => {
    assert.strictEqual(output.results[0].name, 'R2-D2')
  })

Example at: examples/entity-request-options.js

For more examples of request entities, see the Examples, the Integration Examples, and the unit tests: Request Definitions.

Inspecting Request Entities

You may inspect a Request entity through the params.inspect property.

note: At the moment this feature is only available on Request entity, PRs are welcome.

SYNOPSIS

dataPoint.addEntities({
  'request:<entityId>': {
    params: {
      inspect: Boolean|Function
    }
  }
})

Boolean

If params.inspect is true, it will output the entity's information to the console.

Function

If params.inspect is a function, it will be called twice: once before the request is made, and once when the request is resolved. It should have the signature (accumulator: Object, data: Object).

The inspect function is first called just before initiating the request. The first argument is the accumulator, and the second is a data object with these properties:

{
  type: 'request',
  // unique ID that is shared with the 'response' object
  debugId: Number,
  // ex: 'GET'
  method: String,
  // fully-formed URI
  uri: String,
  // the value of request.body (or undefined)
  [body]: String
}

It's then called when the request succeeds or fails. The data object will have a type property of either 'response' or 'error'. The debugId can be used to match the response with the corresponding request.

{
  type: 'response|error',
  // unique ID that is shared with the 'request' object
  debugId: Number,
  // http status code
  statusCode: Number,
}

Hash

A Hash entity transforms a Hash like data structure. It enables you to manipulate the keys within a Hash.

To prevent unexpected results, a Hash can only return Plain Objects, which are objects created by the Object constructor. If a hash resolves to a different type, it will throw an error. This type check occurs before the value is passed to the (optional) outputType reducer.

SYNOPSIS

dataPoint.addEntities({
  'hash:<entityId>': {
    inputType: String | Reducer,
    before: Reducer,
    value: Reducer,
    mapKeys: TransformMap,
    omitKeys: String[],
    pickKeys: String[],
    addKeys: TransformMap,
    addValues: Object,
    compose: ComposeReducer[],
    after: Reducer,
    outputType: String | Reducer,
    error: Reducer,
    params: Object,
  }
})

Properties exposed:

| Key | Type | Description | |:---|:---|:---| | inputType | String, Reducer | type checks the entity's input value, but does not mutate it | | before | Reducer | reducer to be resolved before the entity resolution | | value | Reducer | The value to which the Entity resolves | | mapKeys | Object Reducer | Map to a new set of key/values. Each value accepts a reducer | | omitKeys | String[] | Omits keys from acc.value. Internally. | | pickKeys | String[] | Picks keys from acc.value. Internally. | | addKeys | Object Reducer | Add/Override key/values. Each value accepts a reducer. Internally, this uses the assign reducer helper | | addValues | Object | Add/Override hard-coded key/values. Internally, this uses the assign reducer helper | | compose | ComposeReducer[] | Modify the value of accumulator through an Array of ComposeReducer objects. Think of it as a Compose/Flow Operation, where the result of one operation gets passed to the next one| | after | Reducer | reducer to be resolved after the entity resolution | | outputType | String, Reducer | type checks the entity's output value, but does not mutate it. Collection only supports custom outputType reducers, and not the built-in types like string, number, etc. | | error | Reducer | reducer to be resolved in case of an error | | params | Object | User-defined Hash that will be passed to every reducer within the context of the transform function's execution |

Hash entities expose a set of optional reducers: mapKeys, omitKeys, pickKeys, addKeys, and addValues. When using more than one of these reducers, they should be defined through the compose property.

Hash.value

const input = {
  a: {
    b: {
      c: 'Hello',
      d: ' World!!'
    }
  }
}

dataPoint.addEntities({
  'hash:helloWorld': {
    value: '$a.b'
  }
})

dataPoint
  .resolve('hash:helloWorld', input)
  .then((output) => {
    assert.deepStrictEqual(output, {
      c: 'Hello',
      d: ' World!!'
    })
  })

Example at: examples/entity-hash-context.js

Hash.mapKeys

Maps to a new set of key/value pairs through a object reducer, where each value is a Reducer.

Going back to our GitHub API examples, let's map some keys from the result of a request:

const _ = require('lodash')

dataPoint.addEntities({
  'hash:mapKeys': {
    mapKeys: {
      // map to acc.value.name
      name: '$name',
      // uses a list reducer to
      // map to acc.value.name
      // and generate a string with
      // a function reducer
      url: [
        '$name',
        input => {
          return `https://github.com/ViacomInc/${_.kebabCase(input)}`
        }
      ]
    }
  }
})

const input = {
  name: 'DataPoint'
}

dataPoint.resolve('hash:mapKeys', input).then(output => {
  assert.deepStrictEqual(output, {
    name: 'DataPoint',
    url: 'https://github.com/ViacomInc/data-point'
  })
})

Example at: examples/entity-hash-mapKeys.js

Hash.addKeys

Adds keys to the current Hash value. If an added key already exists, it will be overridden.

Hash.addKeys is very similar to Hash.mapKeys, but the difference is that mapKeys will ONLY map the keys you give it, whereas addKeys will ADD/APPEND new keys to your existing acc.value. You may think of addKeys as an extend operation.

dataPoint.addEntities({
  'hash:addKeys': {
    addKeys: {
      nameLowerCase: ['$name', input => input.toLowerCase()],
      url: () => 'https://github.com/ViacomInc/data-point'
    }
  }
})

const input = {
  name: 'DataPoint'
}

dataPoint.resolve('hash:addKeys', input).then(output => {
  assert.deepStrictEqual(output, {
    name: 'DataPoint',
    nameLowerCase: 'datapoint',
    url: 'https://github.com/ViacomInc/data-point'
  })
})

Example at: examples/entity-hash-addKeys.js

Hash.pickKeys

Picks a list of keys from the current Hash value.

The next example is similar to the previous example. However, instead of mapping key/value pairs, this example just picks some of the keys.

dataPoint.addEntities({
  'hash:pickKeys': {
    pickKeys: ['url']
  }
})

const input = {
  name: 'DataPoint',
  url: 'https://github.com/ViacomInc/data-point'
}

dataPoint.resolve('hash:pickKeys', input).then(output => {
  // notice how name is no longer 
  // in the object
  assert.deepStrictEqual(output, {
    url: 'https://github.com/ViacomInc/data-point'
  })
})

Example at: examples/entity-hash-pickKeys.js

Hash.omitKeys

Omits keys from the Hash value.

This example will only omit some keys, and let the rest pass through:

dataPoint.addEntities({
  'hash:omitKeys': {
    omitKeys: ['name']
  }
})

// notice how name is no longer in the object
const expectedResult = {
  url: 'https://github.com/ViacomInc/data-point'
}

const input = {
  name: 'DataPoint',
  url: 'https://github.com/ViacomInc/data-point'
}

dataPoint.resolve('hash:omitKeys', input).then(output => {
  assert.deepStrictEqual(output, expectedResult)
})

Example at: examples/entity-hash-omitKeys.js

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

DataPoint

Table of Contents

Getting Started

Hello World Example

Fetching remote services

API

create

DataPoint.create example

createReducer

resolve

transform

transform options

resolveFromAccumulator

addEntities

addValue

Accumulator

Reducers

Path Reducer

Root path $

Access accumulator reference

Object Path

Object Map

Function Reducer

Returning a value

Returning a Promise

With a callback parameter

Object Reducer

Entity Reducer

Entity By Id Reducer

List Reducer

Conditionally execute an entity

Reducer Helpers

assign

map

filter

find

constant

parallel

withDefault

Entities

Instance Entity

Registered Entity

Entity Base API

Entity type check

Entity Types

Reducer

Model

Model.value

Model.before

Model.after

Model.error

Model.params

Entry

Request

Request.url

Request.url as StringTemplate

Inspecting Request Entities

Hash

Hash.value

Hash.mapKeys

Hash.addKeys

Hash.pickKeys

Hash.omitKeys