@delucis/reading-data

Aggregate data from different sources.

Installation

npm install --save @delucis/reading-data

Usage

Introduction

@delucis/reading-data provides an interface for gathering data from third-party APIs such as Instapaper.

On its own, this module doesn’t do much but provides a framework for plugins to add support for individual services via its .use() method.

An arbitrary number of plugins can be included and then run in parallel using the .run() method.

const READING_DATA = require('@delucis/reading-data')
const INSTAPAPER_PLUGIN = require('@delucis/reading-data-instapaper')

READING_DATA.use(INSTAPAPER_PLUGIN, {
  // plugin settings
})

READING_DATA.run()

Handling Asynchronous Responses

Often, plugins can take some time to fetch data. For example, they might be sending a request over the network or loading a file from disk. reading-data makes it easy to wait for this data to arrive and then use it.

This can be achieved either by using .then() Promise syntax…

READING_DATA.run().then((result) => {
  console.log(result.data) // prints the gathered data to the console
})

…or by writing your own asynchronous functions.

myAsyncDataLogger = async function () {
  await READING_DATA.run()
  console.log(READING_DATA.data)
}

myAsyncDataLogger()  // prints the gathered data to the console

Hooks during .run()

When you call .run() on a reading-data instance, by default it will cycle through three hooks — preload, fetch, and process — calling each plugin that is configured for that hook. Additional hooks can be registered using the .addHook() method.

If a plugin does not set a default hook to run on, it will be called during the fetch hook.

This can be configured by setting a hooks option for that plugin.

// call myPlugin.data() during the preload hook
READING_DATA.use(myPlugin, {
  hooks: 'preload'
})

Scoping your data

When you call .run() on a reading-data instance, it adds data returned by any plugins in use to its .data property. You set the scope for a plugin in its options object.

READING_DATA.use(myPlugin, {
  scope: 'testData'
})
// returns data to READING_DATA.data.testData

This means you can have multiple plugins working during the same hook, but with separate scopes. If you need to work on the same scope with several plugins, for example in order to first fetch some data and then process it, this must be done in different hooks. Scopes are called in parallel, while hooks are called sequentially.

READING_DATA.use(myFetchPlugin, {
  scope: 'myData',
  hooks: 'fetch'
}).use(myProcessingPlugin, {
  scope: 'myData',
  hooks: 'process'
})

Using a JSONPath to specify scope

Normally you set the scope of a plugin using a string. For example you might fetch information about books you’ve read to 'myBookshelf' and a collection of recipes to 'myMenu'.

What if you had a plugin that converted strings to all caps, and you wanted to store all your book and recipe titles in all caps? You could specify this plugin’s scope using a JSONPath expression, which should always start with a $ character.

READING_DATA.use(uppercaser, {
  scope: ['$.myBookshelf..title', '$.myMenu..title']
})

This would call the uppercaser plugin on every title property that is a child of myBookshelf and every title property that is a child of myMenu. You could even set scope: '$..title', but that might be dangerous if another scope also had title children.

N.B. Because JSONPath is effectively a search mechanism, it requires a data structure to already be in place. For this reason, JSONPath scopes are best suited to situations where you need to process already retrieved data.

Preloading Data

You may have existing data that should be expanded upon or used during the .run() cycle. If so, you can pass it to a reading-data instance using the .preloadData() method.

const READING_DATA = require('@delucis/reading-data')
const EXISTING_DATA = require('./some-data-i-saved-earlier.json')

READING_DATA.preloadData(EXISTING_DATA)

The .preloadData() method can also enable or disable data preloading if passed a boolean:

READING_DATA.preloadData(false) // disables preloading

Preloading data tries to be non-destructive

When the .run() method is called, preloaded data will be added key-by-key to .data using Object.assign. This means it is safe to use .preloadData() on a reading-data instance that is already holding some data as long as you are scoping your data properly.

READING_DATA.use(myPlugin, { scope: 'myPluginScope' })
READING_DATA.run() // adds some data to READING_DATA.data.myPluginScope
READING_DATA.uninstall(myPlugin) // removes the plugin that added data

READING_DATA.preloadData({ myPreloadScope: { /* ... */ }})
READING_DATA.run()
// READING_DATA.data now contains:
// {
//   myPluginScope: { /* ... */ },
//   myPreloadScope: { /* ... */ }
// }

Preloading data only happens once

Data preloading only happens the first time the .run() method is called after using .preloadData(). This prevents the same data being loaded twice and avoids overwriting a scope that may have been updated with newer data by a plugin.

In general this is probably the desired behaviour in a flow that moves from preloading data, to fetching data, to processing data. If you need to re-load data that you had previously preloaded, simply pass true to .preloadData().

let dataToLoad = { myPreloadScope: { text: 'I pre-exist.' } }
READING_DATA.preloadData(dataToLoad)
READING_DATA.run() // adds dataToLoad to READING_DATA.data
READING_DATA.run() // doesn’t try to reload dataToLoad
READING_DATA.preloadData(true)
READING_DATA.run() // adds dataToLoad to READING_DATA.data