@lblod/rdfa-streaming-parser

v2.0.2-custom-html-extractor

Published

a year ago

A fast and lightweight streaming RDFa parser

Downloads

0High
0Medium
0Low

rdfa linked data semantic web rdf rdfjs streaming asynchronous

RDFa Streaming Parser

A fast and lightweight streaming and 100% spec-compliant RDFa 1.1 parser, with RDFJS representations of RDF terms, quads and triples.

The streaming nature allows triples to be emitted as soon as possible, and documents larger than memory to be parsed.

Installation

$ npm install rdfa-streaming-parser

$ yarn add rdfa-streaming-parser

This package also works out-of-the-box in browsers via tools such as webpack and browserify.

Require

import {RdfaParser} from "rdfa-streaming-parser";

const RdfaParser = require("rdfa-streaming-parser").RdfaParser;

Usage

RdfaParser is a Node Transform stream that takes in chunks of RDFa data, and outputs RDFJS-compliant quads.

It can be used to pipe streams to, or you can write strings into the parser directly.

While not required, it is advised to specify the profile of the parser by supplying a contentType or profile constructor option.

Print all parsed triples from a file to the console

const myParser = new RdfaParser({ baseIRI: 'https://www.rubensworks.net/', contentType: 'text/html' });

fs.createReadStream('index.html')
  .pipe(myParser)
  .on('data', console.log)
  .on('error', console.error)
  .on('end', () => console.log('All triples were parsed!'));

Manually write strings to the parser

const myParser = new RdfaParser({ baseIRI: 'https://www.rubensworks.net/', contentType: 'text/html' });

myParser
  .on('data', console.log)
  .on('error', console.error)
  .on('end', () => console.log('All triples were parsed!'));

myParser.write('<?xml version="1.0"?>');
myParser.write(`<!DOCTYPE html>
<html>

<head prefix="foaf: http://xmlns.com/foaf/0.1/">`);
myParser.write(`<link rel="foaf:primaryTopic foaf:maker" href="https://www.rubensworks.net/#me" />`);
myParser.write(`</head>`);
myParser.write(`<body>`);
myParser.write(`</body>`);
myParser.write(`</html>`);
myParser.end();

Import streams

This parser implements the RDFJS Sink interface, which makes it possible to alternatively parse streams using the import method.

const myParser = new RdfaParser({ baseIRI: 'https://www.rubensworks.net/', contentType: 'text/html' });

const myTextStream = fs.createReadStream('index.html');

myParser.import(myTextStream)
  .on('data', console.log)
  .on('error', console.error)
  .on('end', () => console.log('All triples were parsed!'));

Configuration

Optionally, the following parameters can be set in the RdfaParser constructor:

dataFactory: A custom RDFJS DataFactory to construct terms and triples. (Default: require('@rdfjs/data-model'))
baseIRI: An initial default base IRI. (Default: '')
language: A default language for string literals. (Default: '')
vocab: The initial vocabulary. (Default: '')
defaultGraph: The default graph for constructing quads. (Default: defaultGraph())
features: A hash of features that should be enabled. Defaults to the features defined by the profile. (Default: all features enabled)
profile: The RDFa profile to use. (Default: profile with all features enabled)
contentType: The content type of the document that should be parsed. This can be used as an alternative to the 'profile' option. (Default: profile with all features enabled)
htmlParseListener: An optional listener for the internal HTML parse events, should implement IHtmlParseListener (Default: null)

new RdfaParser({
  dataFactory: require('@rdfjs/data-model'),
  baseIRI: 'http://example.org/',
  language: 'en-us',
  vocab: 'http://example.org/myvocab',
  defaultGraph: namedNode('http://example.org/graph'),
  features: { langAttribute: true },
  profile: 'html',
  htmlParseListener: new MyHtmlListener(),
});

Profiles

On top of RDFa Core 1.1, there are a few RDFa variants that add specific sets of rules, which are all supported in this library:

HTML+RDFa 1.1: Internally identified as the 'html' profile with 'text/html' as content type.
XHTML+RDFa 1.1: Internally identified as the 'xhtml' profile with 'application/xhtml+xml' as content type.
SVG Tiny 1.2: Internally identified as the 'xml' profile with 'application/xml', 'text/xml' and 'image/svg+xml' as content types.

This library offers three different ways to define the RDFa profile or setting features:

Content type: Passing a content type such as 'text/html' to the contentType option in the constructor.
Profile string: Passing '', 'core', 'html', 'xhtml' or 'svg' to the profile option in the constructor.
Features object: A custom combination of features can be defined by passing a features option in the constructor.

The table below lists all possible RDFa features and in what profile they are available:

| Feature | Core | HTML | XHTML | XML | Description | | -------------------------------- | ---- |----- | ----- | --- | ----------- | | baseTag | | ✓ | ✓ | | If the baseIRI can be set via the <base> tag. | | xmlBase | | | | ✓ | If the baseIRI can be set via the xml:base attribute. | | langAttribute | | ✓ | ✓ | ✓ | If the language can be set via the language attribute. | | onlyAllowUriRelRevIfProperty | ✓ | ✓ | ✓ | | If non-CURIE and non-URI rel and rev have to be ignored if property is present. | | inheritSubjectInHeadBody | | ✓ | ✓ | | If the new subject can be inherited from the parent object if we're inside <head> or <body> if the resource defines no new subject. | | datetimeAttribute | | ✓ | ✓ | ✓ | If the datetime attribute must be interpreted as datetimes. | | timeTag | | ✓ | ✓ | ✓ | If the <time> tag contents should be interpreted as datetimes. | | htmlDatatype | | ✓ | ✓ | | If rdf:HTML as datatype should cause tag contents to be serialized to text. | | copyRdfaPatterns | ✓ | ✓ | ✓ | | If rdfa:copy property links can refer to rdfa:Pattern's for copying. | | xmlnsPrefixMappings | ✓ | ✓ | ✓ | ✓ | If prefixes should be extracted from xmlns. | | skipHandlingXmlLiteralChildren | | | | | If children of rdf:XMLLiteral should not be handled as RDFa anymore. This is not part of the RDFa spec. | | xhtmlInitialContext | | | ✓ | | If the XHTML initial context should be included in the initial prefixes. | | roleAttribute | | ✓ | ✓ | ✓ | If the role attribute should be handled. |

How it works

This tool makes use of the highly performant htmlparser2 library for parsing HTML in a streaming way. It listens to tag-events, and maintains the required tag metadata in a stack-based datastructure, which can then be emitted as triples as soon as possible.

Our algorithm closely resembles the suggested processing sequence, with a few minor changes to make it work in a streaming way.

If you want to make use of a different HTML/XML parser, you can create a regular instance of RdfaParser, and just call the following methods yourself directly:

onTagOpen(name: string, attributes: {[s: string]: string})
onText(data: string)
onTagClose()

Specification Compliance

This parser passes all tests from the RDFa 1.1 test suite. More specifically, the following manifests are explicitly tested:

HTML+RDFa 1.1 (HTML4)
HTML+RDFa 1.1 (HTML5)
HTML+RDFa 1.1 (XHTML5)
SVGTiny+RDFa 1.1
XHTML+RDFa 1.1
XML+RDFa 1.1

The following optional features for RDFa processors are supported:

Processing the @role attribute.

The following optional features for RDFa processors are not supported (yet):

License

This software is written by Ruben Taelman.

This code is released under the MIT license.