LDES Substring Bucketizer

The purpose of the substring bucketizer is to apply a substring fragmentation tot LDES members based on a property path.

An LDES bucketizer adds triples with the ldes bucket predicate (https://w3id.org/ldes#bucket) to the array of quads representating an LDES member, indicating the bucket in which the member belongs.

The fragmentation strategy is based on previous work and the following method is applied:

Example

We assume the following LDES member on which a substring fragmentation with the property path set to (<http://www.w3.org/2000/01/rdf-schema#label>) will be applied.

<http://example.ord/id/123@456> dct:created "2002-08-13T16:33:18+02:00"^^xsd:dateTime ;
          dct:isVersionOf <http://example.org/id/123> ;
          prov:generatedAtTime "2021-09-07T15:44:05.975Z"^^xsd:dateTime ;
          rdfs:label "John Doe" .

After passing through the bucketizer, the LDES member will have extra triples:

<http://example.ord/id/123@456> dct:created "2002-08-13T16:33:18+02:00"^^xsd:dateTime ;
          dct:isVersionOf <http://example.org/id/123> ;
          prov:generatedAtTime "2021-09-07T15:44:05.975Z"^^xsd:dateTime ;
          rdfs:label "John Doe" ;
          ldes:bucket "j", "jo", "joh", "john", "d", "do", "doe" .

Install

> npm i @treecg/ldes-substring-bucketizer

Usage

A bucketizer should always be used in combination with the LDES client. More information on how to setup an LDES client can be found here. It is important to set the option in the LDES client to receive the LDES member as an array of quads: representation: 'quads'.

The bucketizer expects a property path as input and should have the following structure (<predicate1> <predicate2> ...)

import { SubstringBucketzer } from '@treecg/ldes-substring-bucketizer'

const run = async (): Promise<void> => {
  const options = {...};
  const url = ...;

  const bucketizer = await SubstringBucketizer.build('(<http://www.w3.org/2000/01/rdf-schema#label>)');

  const ldes = LDESClient.createReadStream(url, options);
  ldes.on('data', (member) => {
    bucketizer.bucketize(member.quads, member.id)

    // Continue processing the member, but now the array of quads will have an extra triples, the bucket triples
  });
}

run().catch(error => console.error(error.stack));