harvesting-utils

Utils library for harvesting RPDE feeds.

Version 0.X.X

This library is currently in version 0.X.X, which means that the API will not be stable until 1.0.0.

Install

This library can be installed as an npm package using the following command:

$ npm install git://github.com/openactive/harvesting-utils.git

Usage

const { harvestRPDE } = require('@openactive/harvesting-utils')

harvestRPDE({
  baseUrl: '...',
  /* ...relevant parameters here */
});

Examples

A very simple example of harvestRPDE can be found in examples/simple-rpde-harvester.js. For more information on this script see here.

API Reference

harvestRPDE

Indefinitely harvests an RPDE feed, following the "expected consumer behaviour" described in the RPDE spec.

N.B. This function will run indefinitely, and only return if a fatal error occurs. For this reason, you will generally not want to run await harvestRPDE(..).

Required Parameters

| Parameter | Type | Description | | --------- | ---- | ----------- | | baseUrl | string | Feed URL to harvest | | feedContextIdentifier | string | Unique identifier for feed within the dataset eg ScheduledSession | | headers | () => Promise<Object.<string,string>> | Function that returns headers needed to make a request to the feed URL | | processPage | (rpdePage: any, feedIdentifier: string, isInitialHarvestComplete: () => boolean) => Promise | Function that processes items in each page of the feed | | onFeedEnd | () => Promise | Function that is called when the last page of the feed is reached. This function may be called multiple times if new items are added after the first time harvestRPDE() reaches the last page | | onError | () => Promise | Function that is called if the harvest errors | | isOrdersFeed | boolean | Is the feed an Orders feed? |

Optional Parameters

| Parameter | Type | Description | | --------- | ---- | ----------- | | state | object | Existing state can be passed in and manipulated within harvestRPDE() | | state.context | FeedContext | Context about the feed. Default: new FeedContext(feedContextIdentifier,baseUrl, multibar) | | state.feedContextMap | Map<string, FeedContext> | Map containing FeedContexts about this and other feeds within the dataset. Default: new Map() | | state.startTime | Date | Start time of the harvest. Default: new Date() | | loggingFns | object | Logging functions for different cases | | loggingFns.log | (message?: any, ...optionalParams: any[]) => void | Normal logging. Default: console.log | | loggingFns.logError | (message?: any, ...optionalParams: any[]) => void | Error logging. Default: console.error | | loggingFns.logErrorDuringHarvest | (message?: any, ...optionalParams: any[]) => void | Error logging during the harvest Default: console.error | | config| object | Configuration options | | config.howLongToSleepAtFeedEnd | () => number | How long to wait, in milliseconds, before re-polling a feed after fetching the last page (RPDE spec). Default: () => 500 | | config.WAIT_FOR_HARVEST | boolean | Whether to wait for harvest to complete and run onFeedEnd() function. Default: true | | config.VALIDATE_ONLY | boolean | TODO. Default: false | | config.VERBOSE | boolean | Verbose logging. Default: false | | config.ORDER_PROPOSALS_FEED_IDENTIFIER | string | TODO. Default: null | | config.REQUEST_LOGGING_ENABLED | boolean | Extra logging around the request. Default: false | | options | object | Optional features | | options.multibar | import('cli-progress').MultiBar | If using cli-progress.Multibar, this can be supplied and harvesting updates will be provided to the multibar. Default: null | | options.pauseResume | {waitIfPaused: () => Promise} | Function, if implemented, that can be used to pause harvesting. Default: null |

createFeedContext

Function that creates a FeedContext object

Required Parameters

| Parameter | Type | Description | | --------- | ---- | ----------- | | feedContextIdentifier | string | Unique identifier for feed within the dataset eg ScheduledSession | | baseUrl | string | Feed URL to harvest |

Optional Parameters

| Parameter | Type | Description | | --------- | ---- | ----------- | | multibar | import('cli-progress').MultiBar | If using cli-progress.Multibar, this can be supplied and context values will be provided to the multibar. Default: null |

progressFromContext

Function that returns harvesting progress values from a FeedContext object

Required Parameters

| Parameter | Type | Description | | --------- | ---- | ----------- | | context | FeedContext | FeedContext object to get progress values from |

harvestRpdeLossless

harvestRpdeLossless has the same function signature as harvestRpde. However it is capable of handling modified values that are too large for JavaScript numbers to handle natively ie > 2^53. This is handled by storing them as strings in memory.

For more guidance on how to handle these values, see here.

Developing

TypeScript

The code is written in native JS, but uses TypeScript to check for type errors. TypeScript uses JSDoc annotations to determine types (See: Type Checking JavaScript Files) from our native .js files.

In order for these types to be used by other projects, they must be saved to TypeScript Declaration files. This is enabled by our tsconfig.json, which specifies that declaration files are to be generated and saved to built-types/ (As an aside, the reason that the package's types must be saved to .d.ts files is due to TypeScript not automatically using JS defined types from libraries. There is a good reason for this and proposals to allow it to work at least for certain packages. See some of the discussion here: https://github.com/microsoft/TypeScript/issues/33136).

For this reason, TypeScript types should be generated after code changes to make sure that consumers of this library can use the new types. The openactive-test-suite project does this automatically in its pre-commit hook, which calls npm run gen-types

TypeScript-related scripts:

• check-types: This uses the tsconfig.check.json config, which does not emit any TS declaration files - all it does is check that there are no type errors. This is used for code tests.

• gen-types: This uses the tsconfig.gen.json config, which emits TS declaration files into built-types/.

  Additionally, it copies programmer-created .d.ts files from our source code (e.g. src/types/Criteria.d.ts) into built-types/. This is because our code references these types, so they must be in the built-types/ directory so that the relative paths match (e.g. so that import('../types/Criteria').Criteria works).