jbr.js – Just a Benchmark Runner

A simple tool to initialize benchmarking experiments, run them, and analyze their results.

Experiments that are created and executed with this tool are fully reproducible, as experiments are fully deterministic, and metadata on all exact installed dependency versions is emitted together with the results.

Guides

• Setting up a single Linked Data Fragments experiment
• Setting up a factorial Linked Data Fragments experiment

Requirements

• Node.js (1.12 or higher)

For certain experiment types, you may also require Docker.

Installation

$ npm install -g jbr

or

$ yarn global add jbr

Usage

This tool offers commands for executing the whole experimentation chain:

1. Initialization: Create a new experiment. This should be done only once.
2. Data Preparation: Generating a dataset and query set. This should be done only once.
3. Running Experiments: Starting the required machines and running the benchmark.

Full usage:

jbr <command>

Commands:
  jbr clean                      Cleans up an experiment
  jbr generate-combinations      Generate combinations of experiment templates
  jbr init <type> <name>         Initializes a new experiment
  jbr pack                       Create an archive of the experiment output
  jbr prepare                    Prepare data for the current experiment
  jbr run                        Run the current experiment
  jbr set-hook <hook> <handler>  Provide a handler for a hook in an experiment
  jbr validate                   Validate the current experiment

Options:
      --version         Show version number                            [boolean]
      --cwd             The current working directory      [string] [default: .]
  -m, --mainModulePath  Path from which modules should be loaded        [string]
  -v, --verbose         If more logging output should be generated     [boolean]
  -d, --dockerOptions   Path to a file with custom Docker options       [string]
  -b, --breakpoints     If experiment breakpoints are enabled          [boolean]
      --help            Show help

1. Initialization

$ jbr init experiment-type my-experiment
$ cd my-experiment

Running this command will initialize a new experiment of the given type (experiment-type) in a new directory of the provided experiment name (my-experiment).

The experiment type must exist on npm under the @jbr-experiment/* scope. Click here for a full list of available experiment types. For example, the watdiv experiment can be used because the @jbr-experiment/watdiv package exists on npm.

The created directory will contain all default required files for running an experiment. You can initialize this directory as a git repository.

In most cases, you will have to configure at least one hook handler for your experiment, such as defining the SPARQL query engine you want to evaluate for a given benchmark experiment. Furthermore, you will usually need to edit the jbr-experiment.json file to configure your experiment.

2. Data Preparation

In order to run all preprocessing steps, such as creating all required datasets, invoke the prepare step:

$ jbr prepare

All prepared files will be contained in the generated/ directory.

When running this command, existing files within generated/ will not be overwritten by default. These can be forcefully overwritten by passing the -f option.

3. Running Experiments

Once the experiment has been fully configured and prepared, you can run it:

$ jbr run

Once the run step completes, results will be present in the output/ directory.

Configurability

All experiments will have a jbr-experiment.json in which the properties of an experiment can be set. The parameters of such a config file are dependent on the type of experiment that is being initialized.

Depending on the experiment type, you may also need to change certain files within the input/ directory.

Hooks

Most experiment types expose certain hooks, which allow you to plug in certain hook handlers. For example, the WatDiv experiment type exposes the hookSparqlEndpoint hook. This hook is used to plug in a certain SPARQL query engine, which is what WatDiv will use to run its benchmark over.

Hook handler types must exist on npm under the @jbr-hook/* scope. Click here for a full list of available hook handler types. For example, the sparql-endpoint-comunica hook handler can be used because the @jbr-hook/sparql-endpoint-comunica package exists on npm.

Directory structure

A jbr experiment typically has the following directory structure:

my-experiment/
  .gitignore
  jbr-experiment.json  # Main config of your experiment
  package.json
  generated/           # Prepared data files
  input/               # More indirect configuration
  output/              # Raw output of the experiment
  node_modules/

To enable reproducibility, it is highly recommended to place these experiments under version control, e.g. via a git repository.

The following files and directories do not have to be added to this repository, as they are derived and can be reproduced:

my-experiment/
  generated/
  output/
  node_modules/

Advanced

Combinations-based Experiments

Certain experiments may be designed to compare the effect different factors to each other, such as full factorial experiments, or fractional experiments. For instance, this may be used to compare the effect of running a certain system once with algorithm A and once with B, and measuring the effects.

Using jbr, you can easily setup and handle such combination-based experiments as follows:

1. Initialize experiment

Experiments that should be combinations-based must be initialized using the -c flag:

$ jbr init -c experiment-type my-experiment
$ cd my-experiment

Instead of creating a jbr-experiment.json file, this will create a jbr-experiment.json.template file, together with an accompanying jbr-combinations.json file.

2. Define combinations

Inside the jbr-experiment.json.template file (and input text files), you may define any number of variables using the %FACTOR-variableName% syntax. Inside the jbr-combinations.json file, you can define corresponding values for the given variables.

For example, jbr-experiment.json.template can look like:

{
  "@context": [
    "https://linkedsoftwaredependencies.org/bundles/npm/jbr/^2.0.0/components/context.jsonld",
    "https://linkedsoftwaredependencies.org/bundles/npm/@jbr-experiment/ldbc-snb-decentralized/^1.0.0/components/context.jsonld",
    "https://linkedsoftwaredependencies.org/bundles/npm/@jbr-hook/sparql-endpoint-comunica/^1.0.0/components/context.jsonld"
  ],
  "@id": "urn:jrb:experimentname",
  "@type": "MyExperiment",
  "cpu_percentage": %FACTOR-cpu%,
  "memory_percentage": %FACTOR-memory%
}

Variable values can be assigned in jbr-combinations.json:

{
  "@context": [
    "https://linkedsoftwaredependencies.org/bundles/npm/jbr/^2.0.0/components/context.jsonld"
  ],
  "@id": "urn:jrb:experimentname-combinations",
  "@type": "FullFactorialCombinationProvider",
  "commonGenerated": false,
  "factors": {
    "cpu": [ 50, 100 ],
    "memory": [ 50, 100 ]
  }
}

Because FullFactorialCombinationProvider is used in jbr-combinations.json, all combinations (4) of the cpu and memory variable will apply to this experiment.

If the generated directory can be reused across combinations, then commonGenerated can be set to true.

FractionalCombinationProvider may also be used if only select combinations should apply.

3. Regenerate combinations

Each time you make a change inside your input files, jbr-combinations.json, or jbr-experiment.json.template, you should (re)generate the instantiated combinations by running the following command:

$ jbr generate-combinations

This will create a new combinations/ directory, containing sub-directories for all experiment combinations. Files in this directory should not be modified manually, but should only be managed via the template files and jbr generate-combinations.

4. Handle like any other experiment

From this point on, you can manage this combinations-based experiment like any other jbr experiment.

Concretely, jbr prepare will prepare all combinations, and jbr run will also run all combinations.

If you just want to run a single combination, you can specify its combination id via the -c option:

$ jbr run -c 3

Docker Resource Constraint

Some experiments or hooks may be executed in Docker containers. For these cases, jbr exposes a reusable helper component for defining Docker resource constraints.

For example, the following experiment is configured to use at most 90% of the CPU, and 10MB of memory.

{
  "@type": "SomeExperiment",
  "resourceConstraints": {
    "@type": "StaticDockerResourceConstraints",
    "cpu_percentage": 90,
    "memory_limit": "10m"
  }
}

All possible parameters (all are optional):

• cpu_percentage: Percentage (0-100) of the total CPU power that can be used. E.g. when fully consuming 4 cores, this value must be set to 100.
• memory_limit: Memory usage limit, e.g. '10m', '1g'.

Running against a different Docker instance

By default, Docker-based experiment will look for and use the Docker installation on your local machine. In some cases, you may want to run experiments within remote Docker instances. In such cases, you can use the -d or --dockerOptions option to pass a custom Docker options file.

For example, Docker options can be set when running an experiment as follows:

$ jbr run -d docker-options.json

docker-options.json for the default socket:

{
    "socketPath": "/var/run/docker.sock"
}

docker-options.json for running against a different host:

{
    "host": "http://192.168.1.10",
    "port": 3000
}

More configuration options can be found at https://github.com/apocas/dockerode#getting-started

License

jbr.js is written by Ruben Taelman.

This code is copyrighted by Ghent University – imec and released under the MIT license.