live-source

Provide an object (a "doc") which shows the state of a web resource, updated and emitting 'change' events when the web resource changes. This core module just uses polling, but related modules act as plugins which can handle various kinds of pub/sub in the cases where they are implemented.

Ordinary usage

Here is example1.js. It polls a resource that returns the unix time, changing every second. It misses some values because it's polling at the default rate of 1.5s. See example1a for faster polling, although that's not usually recommended.

const livesource = require('live-source')

const url = 'https://time.hawkeworks.com/?units=1'
const doc = livesource.open(url)
let count = 0
doc.on('change', ({text}) => {
  console.log(++count, text)
  if (count >= 5) doc.close()
})
/* 
=> 
1 '1561761213'
2 '1561761215'
3 '1561761216'
4 '1561761218'
5 '1561761219'
*/

Adding formats with custom parsers

Custom formats can be added with addFormat, which takes an {init, run} object, with two functions:

• init(doc) sets properties of doc at constructor time, for what you want the initial values to be for the properties this format produces. The init value should usually be something the source would never have, so that a change event fires at the first load, while also being harmless.

• run(doc) is called after doc.text has been set by fetch(). It should read properties of the doc, like doc.text, and set other properties appropriately. If any format's run() function sets doc.changed to true or false, that determines whether the essential value of the doc is considered changed, and thus whether a 'change' event fires.

Example2 uses two formats, the first turning doc.text into a parsed integer, and the second turning that parsed integer into a Date object, assuming it's a unix timestamp.

const livesource = require('live-source')

const int = {
  init: doc => {
    doc.intValue = 0
  },
  run: doc => {
    const old = doc.intValue
    doc.intValue = parseInt(doc.text)
    return old !== doc.intValue
  }
}

const unixtime = {
  init: doc => {
    doc.time = new Date(0)
  },
  run: doc => {
    const old = doc.time
    doc.time = new Date(doc.intValue * 1000)
    return old.getTime() !== doc.time.getTime()
  }
}

const url = 'https://time.hawkeworks.com/?units=1'
livesource.addFormat(int)
livesource.addFormat(unixtime)
const doc = livesource.open(url)
let count = 0
doc.on('change', ({text, intValue, time}) => {
  console.log(++count, {text, intValue, time})
  if (count >= 3) doc.close()
})
/* 
=>
1 { text: '1561759480',
  intValue: 1561759480,
  time: 2019-06-28T22:04:40.000Z }
2 { text: '1561759482',
  intValue: 1561759480,
  time: 2019-06-28T22:04:40.000Z }
3 { text: '1561759483',
  intValue: 1561759480,
  time: 2019-06-28T22:04:40.000Z }
*/

Adding openers

The livesource module is an instance of "Opener", and has the open(url) function you've seen above. You can give it sub-openers for handling specific URLs, based on looking at their text. Sub-openers have to decide decide synchronously if they'll handle each URL, because open returns synchronously.

An example of sub-openers is in live-source-gdoc, which does special handling for Google Doc URLs.

todo: allow these handlers to take over after the first fetch, since that might be when we discover the page has a websub hub.

Additional modules

• live-source-proxy is useful for lightweight clients, aggregating polling into hubs, or clients in a browser.