RDF-Test-Suite.js

This tool executes the RDF and SPARQL test suites with any given system.

It can either output human-readable test results to the console, or it can output machine-readable reports in the EARL vocabulary.

Currently, the following test suites are supported:

• SPARQL 1.0
• SPARQL 1.1
• RDF/XML
• N-Triples
• N-Quads
• Turtle
• TriG
• JSON-LD (1.0 and 1.1)
• RDFa
• Microdata to RDF
• Notation3
• RDF-star and SPARQL-star
• JSON-LD-Star

Not all RDF test suites are supported at the moment. However, this package is fully modular, so that support for different test cases can be implemented easily.

Planned support:

• SPARQL 1.1 (other specifications)
• RDF Schema and Semantics
• JSON-LD (other test suites)

Installation

Either install it globally:

$ yarn global add rdf-test-suite

Or locally (as a dev dependency):

$ yarn add --dev rdf-test-suite

Usage

After installing, the rdf-test-suite script will become available.

This script requires some kind of engine as first argument, and a test suite manifest URL as second argument.

The engine is a JavaScript file that can handle the test suite. The interface of this engine depends on the manifest. For example, the SPARQL 1.1 test suite requires an IQueryEngine, while the RDF/XML test suite requires a IParser.

Basic execution

The following command executes the SPARQL 1.1 test suite on the engine myengine.js:

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl

An example of what this myengine.js looks like is available for the Comunica SPARQL engine.

This command will output something like this:

...
✔ syn-bad-pname-04 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_04) 0.293588ms
✔ syn-bad-pname-05 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_05) 0.498646ms
✔ syn-bad-pname-06 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_06) 0.513092ms
✔ syn-bad-pname-07 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_07) 0.498646ms
✔ syn-bad-pname-08 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_08) 0.293588ms
✔ syn-bad-pname-09 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_09) 0.293588ms
✔ syn-bad-pname-10 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_10) 0.516346ms
✔ syn-bad-pname-11 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_11) 0.498646ms
✔ syn-bad-pname-12 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_12) 0.516346ms
✔ syn-bad-pname-13 (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pn_bad_13) 0.498646ms
✔ syn-pp-in-collection (http://www.w3.org/2009/sparql/docs/tests/data-sparql11/syntax-query/manifest#test_pp_coll) 0.516346ms
✖ 105 / 268 tests succeeded!

Test filtering

With the -t option, you can filter tests that should be executed based on a regex that will be matched with the test URI.

For example:

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -t test_pn_bad_0

Test skipping

With the --skip option you can filter tests that should be skipped based on a regex that will be matched with the test URI.

For example:

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  --skip test_pn_bad_0

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  --skip (test_pn_bad_0|test_pn_bad_1)

Summarized output

By default, the printed test results are verbose, and print expanded views of failed tests.

You can change this to a more compact view using -o summary.

For example:

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -o summary

EARL output

Test results are human readable by default. This can be changed to output machine-readable reports in the EARL vocabulary in the Turtle serialization. These reports can be published to report the compliance of engines to certain specifications, like on W3C’s RDF Test Curation Community Group.

As EARL reports require some metadata on your engine, you will need to provide a properties file via the -p argument.

This properties file can look something like this:

{
  "applicationBugsUrl": "https://github.com/comunica/comunica/issues",
  "applicationDescription": "A Comunica engine for SPARQL query evaluation over heterogeneous interfaces",
  "applicationHomepageUrl": "http://comunica.linkeddatafragments.org/",
  "applicationNameFull": "Comunica SPARQL",
  "applicationNameNpm": "@comunica/actor-init-sparql",
  "applicationUri": "https://www.npmjs.com/package/@comunica/actor-init-sparql",
  "authors": [
    {
      "homepage": "https://www.rubensworks.net/",
      "name": "Ruben Taelman",
      "uri": "https://www.rubensworks.net/#me"
    }
  ],
  "licenseUri": "http://opensource.org/licenses/MIT",
  "reportUri": null,
  "specificationUris": [
      "http://www.w3.org/TR/sparql11-query/"
  ],
  "version": "1.2.3"
}

The properties file can point to a non-existing file, in which case it will be auto-generated from the package.json file in the current working directory.

For example:

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -o earl -p earl-meta.json

Restricting execution to certain specifications

Some test suites (like the one of SPARQL 1.1) contains tests for multiple specifications. If you only want to test your system for a single specification, then you can define this with the -s parameter.

For example, the following command executes the SPARQL 1.1 test cases that apply to the http://www.w3.org/TR/sparql11-query/ specification.

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -s http://www.w3.org/TR/sparql11-query/

Enabling HTTP caching

By default, rdf-test-suite will look up all required manifest files. If this takes too much time, or if you don't have internet connection, then you can enable HTTP caching with the -c argument, so that all files only have to be looked up once.

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -c path/to/cache/

If you don't provide a caching value after the -c, then the directory will default to .rdf-test-suite-cache/.

Ignore exit code

When there are failing tests, rdf-test-suite will exit with code 1 instead of 0. This can be useful in Continuous Integration tools.

If you want to disable this behaviour, you can add the -e flag to force an exit code 0.

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -e

Pass custom options to the engine

With the optional -i option, a JSON string with arguments can be passed to the engine. These arguments will become available in both IQueryEngine, and IParser.

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -i '{ "myProperty": "myValue" }'

Test timeouts

By default, all tests are allowed to run 3000 ms. Using the -d option, you can change this value.

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -d 5000

Map URLs to local files

In cases where you are developing your own test manifests, it may be useful to intercept certain URL lookups and return local files instead.

In order to achieve this, the -m option can be used, with a mapping defined with the following pattern: URL~PATH.

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -m http://w3c.github.io/rdf-tests/sparql11/data-sparql11/~/path/to/my/files/

Using rejected tests

In cases where you wish to execute a test suite that contains rdft:Rejected test entries rdf-test-suite will skip those tests by default. If you wish to execute those tests as well, you can use the -r option.

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -r

Only using explicitly approved tests

In cases where you wish to execute a test suite that contains rdft:Approved test entries rdf-test-suite will, by default execute those tests as well as all other tests that are not explicitly rejected. If you wish to only execute those tests that are explicitly approved, you can use the -a option.

$ rdf-test-suite myengine.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl \
  -a

Supported test suites

| Manifest | Specification | Interface | Entry manifest | | -------- | ------------- | --------- | -------------- | | SPARQL 1.0 tests | SPARQL 1.0 | IQueryEngine | https://w3c.github.io/rdf-tests/sparql11/data-r2/manifest.ttl | | SPARQL 1.1 tests | SPARQL 1.1 Query | IQueryEngine | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl | | SPARQL 1.1 tests | SPARQL 1.1 Update | IUpdateEngine | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl | | SPARQL 1.1 tests | SPARQL 1.1 Results CSV/TSV | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl | | SPARQL 1.1 tests | SPARQL 1.1 Results JSON | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl | | SPARQL 1.1 tests | SPARQL 1.1 Federated Query | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl | | SPARQL 1.1 tests | SPARQL 1.1 Entailment | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl | | SPARQL 1.1 tests | SPARQL 1.1 Service Description | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl | | SPARQL 1.1 tests | SPARQL 1.1 Protocol | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl | | SPARQL 1.1 tests | SPARQL 1.1 HTTP RDF Update | ✖ | http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl | | RDF/XML Syntax Tests | RDF 1.1 XML Syntax | IParser | http://w3c.github.io/rdf-tests/rdf-xml/manifest.ttl | | N-Triples Tests | RDF 1.1 N-Triples | IParser | http://w3c.github.io/rdf-tests/ntriples/manifest.ttl | | N-Quads Tests | RDF 1.1 N-Quads | IParser | http://w3c.github.io/rdf-tests/nquads/manifest.ttl | | Turtle Tests | RDF 1.1 Turtle | IParser | http://w3c.github.io/rdf-tests/turtle/manifest.ttl | | TriG Tests | RDF 1.1 TriG | IParser | http://w3c.github.io/rdf-tests/trig/manifest.ttl | | JSON-LD Test Suite | JSON-LD (1.0 and 1.1) | IParser | https://w3c.github.io/json-ld-api/tests/toRdf-manifest.jsonld | | JSON-LD Test Suite | JSON-LD (1.0 and 1.1) | IParser | https://w3c.github.io/json-ld-api/tests/html-manifest.jsonld | | JSON-LD Test Suite | JSON-LD (1.0 and 1.1) | ISerializer | https://w3c.github.io/json-ld-api/tests/fromRdf-manifest.jsonld | | JSON-LD-Star Test Suite | JSON-LD-Star | IParser | https://json-ld.github.io/json-ld-star/tests/toRdf-manifest.jsonld | | JSON-LD-Star Test Suite | JSON-LD-Star | ISerializer | https://json-ld.github.io/json-ld-star/tests/fromRdf-manifest.jsonld | | RDFa Test Suite | RDFa 1.1 | IParser | http://rdfa.info/test-suite/test-cases/rdfa1.1/html5/manifest.ttl | | Microdata to RDF Test Suite | Microdata to RDF | IParser | https://w3c.github.io/microdata-rdf/tests/manifest.ttl | | Notation3 Test Suite | N3 Grammar | IParser | https://w3c.github.io/N3/tests/N3Tests/manifest-parser.ttl | | Notation3 Test Suite | Extended N3 Grammar | IParser | https://w3c.github.io/N3/tests/N3Tests/manifest-extended.ttl | | Turtle-star Evaluation Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/turtle/eval/manifest.jsonld | | Turtle-star Syntax Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/turtle/syntax/manifest.jsonld | | TriG-star Evaluation Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/trig/eval/manifest.jsonld | | TriG-star Syntax Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/trig/syntax/manifest.jsonld | | N-Triples-star Syntax Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/nt/syntax/manifest.jsonld | | RDF-star Semantics tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/semantics/manifest.jsonld | | SPARQL-star Syntax Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/sparql/syntax/manifest.jsonld | | SPARQL-star Evaluation Tests | RDF-star and SPARQL-star | ✖ | https://w3c.github.io/rdf-star/tests/sparql/eval/manifest.jsonld |

License

This software is written by Ruben Taelman.

This code is released under the MIT license.