npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

jbr

v5.4.0

Published

Just a Benchmark Runner

Downloads

2,182

Readme

jbr.js – Just a Benchmark Runner

Build status Coverage Status npm version

A simple tool to initialize benchmarking experiments, run them, and analyze their results.

Experiments that are created and executed with this tool are fully reproducible, as experiments are fully deterministic, and metadata on all exact installed dependency versions is emitted together with the results.

Guides

Requirements

For certain experiment types, you may also require Docker.

Installation

$ npm install -g jbr

or

$ yarn global add jbr

Usage

This tool offers commands for executing the whole experimentation chain:

  1. Initialization: Create a new experiment. This should be done only once.
  2. Data Preparation: Generating a dataset and query set. This should be done only once.
  3. Running Experiments: Starting the required machines and running the benchmark.

Full usage:

jbr <command>

Commands:
  jbr clean                      Cleans up an experiment
  jbr generate-combinations      Generate combinations of experiment templates
  jbr init <type> <name>         Initializes a new experiment
  jbr pack                       Create an archive of the experiment output
  jbr prepare                    Prepare data for the current experiment
  jbr run                        Run the current experiment
  jbr set-hook <hook> <handler>  Provide a handler for a hook in an experiment
  jbr validate                   Validate the current experiment

Options:
      --version         Show version number                            [boolean]
      --cwd             The current working directory      [string] [default: .]
  -m, --mainModulePath  Path from which modules should be loaded        [string]
  -v, --verbose         If more logging output should be generated     [boolean]
  -d, --dockerOptions   Path to a file with custom Docker options       [string]
  -b, --breakpoints     If experiment breakpoints are enabled          [boolean]
      --help            Show help

1. Initialization

$ jbr init experiment-type my-experiment
$ cd my-experiment

Running this command will initialize a new experiment of the given type (experiment-type) in a new directory of the provided experiment name (my-experiment).

The experiment type must exist on npm under the @jbr-experiment/* scope. Click here for a full list of available experiment types. For example, the watdiv experiment can be used because the @jbr-experiment/watdiv package exists on npm.

The created directory will contain all default required files for running an experiment. You can initialize this directory as a git repository.

In most cases, you will have to configure at least one hook handler for your experiment, such as defining the SPARQL query engine you want to evaluate for a given benchmark experiment. Furthermore, you will usually need to edit the jbr-experiment.json file to configure your experiment.

2. Data Preparation

In order to run all preprocessing steps, such as creating all required datasets, invoke the prepare step:

$ jbr prepare

All prepared files will be contained in the generated/ directory.

When running this command, existing files within generated/ will not be overwritten by default. These can be forcefully overwritten by passing the -f option.

3. Running Experiments

Once the experiment has been fully configured and prepared, you can run it:

$ jbr run

Once the run step completes, results will be present in the output/ directory.

Configurability

All experiments will have a jbr-experiment.json in which the properties of an experiment can be set. The parameters of such a config file are dependent on the type of experiment that is being initialized.

Depending on the experiment type, you may also need to change certain files within the input/ directory.

Hooks

Most experiment types expose certain hooks, which allow you to plug in certain hook handlers. For example, the WatDiv experiment type exposes the hookSparqlEndpoint hook. This hook is used to plug in a certain SPARQL query engine, which is what WatDiv will use to run its benchmark over.

Hook handler types must exist on npm under the @jbr-hook/* scope. Click here for a full list of available hook handler types. For example, the sparql-endpoint-comunica hook handler can be used because the @jbr-hook/sparql-endpoint-comunica package exists on npm.

Directory structure

A jbr experiment typically has the following directory structure:

my-experiment/
  .gitignore
  jbr-experiment.json  # Main config of your experiment
  package.json
  generated/           # Prepared data files
  input/               # More indirect configuration
  output/              # Raw output of the experiment
  node_modules/

To enable reproducibility, it is highly recommended to place these experiments under version control, e.g. via a git repository.

The following files and directories do not have to be added to this repository, as they are derived and can be reproduced:

my-experiment/
  generated/
  output/
  node_modules/

Advanced

Combinations-based Experiments

Certain experiments may be designed to compare the effect different factors to each other, such as full factorial experiments, or fractional experiments. For instance, this may be used to compare the effect of running a certain system once with algorithm A and once with B, and measuring the effects.

Using jbr, you can easily setup and handle such combination-based experiments as follows:

1. Initialize experiment

Experiments that should be combinations-based must be initialized using the -c flag:

$ jbr init -c experiment-type my-experiment
$ cd my-experiment

Instead of creating a jbr-experiment.json file, this will create a jbr-experiment.json.template file, together with an accompanying jbr-combinations.json file.

2. Define combinations

Inside the jbr-experiment.json.template file (and input text files), you may define any number of variables using the %FACTOR-variableName% syntax. Inside the jbr-combinations.json file, you can define corresponding values for the given variables.

For example, jbr-experiment.json.template can look like:

{
  "@context": [
    "https://linkedsoftwaredependencies.org/bundles/npm/jbr/^2.0.0/components/context.jsonld",
    "https://linkedsoftwaredependencies.org/bundles/npm/@jbr-experiment/ldbc-snb-decentralized/^1.0.0/components/context.jsonld",
    "https://linkedsoftwaredependencies.org/bundles/npm/@jbr-hook/sparql-endpoint-comunica/^1.0.0/components/context.jsonld"
  ],
  "@id": "urn:jrb:experimentname",
  "@type": "MyExperiment",
  "cpu_percentage": %FACTOR-cpu%,
  "memory_percentage": %FACTOR-memory%
}

Variable values can be assigned in jbr-combinations.json:

{
  "@context": [
    "https://linkedsoftwaredependencies.org/bundles/npm/jbr/^2.0.0/components/context.jsonld"
  ],
  "@id": "urn:jrb:experimentname-combinations",
  "@type": "FullFactorialCombinationProvider",
  "commonGenerated": false,
  "factors": {
    "cpu": [ 50, 100 ],
    "memory": [ 50, 100 ]
  }
}

Because FullFactorialCombinationProvider is used in jbr-combinations.json, all combinations (4) of the cpu and memory variable will apply to this experiment.

If the generated directory can be reused across combinations, then commonGenerated can be set to true.

FractionalCombinationProvider may also be used if only select combinations should apply.

3. Regenerate combinations

Each time you make a change inside your input files, jbr-combinations.json, or jbr-experiment.json.template, you should (re)generate the instantiated combinations by running the following command:

$ jbr generate-combinations

This will create a new combinations/ directory, containing sub-directories for all experiment combinations. Files in this directory should not be modified manually, but should only be managed via the template files and jbr generate-combinations.

4. Handle like any other experiment

From this point on, you can manage this combinations-based experiment like any other jbr experiment.

Concretely, jbr prepare will prepare all combinations, and jbr run will also run all combinations.

If you just want to run a single combination, you can specify its combination id via the -c option:

$ jbr run -c 3

Docker Resource Constraint

Some experiments or hooks may be executed in Docker containers. For these cases, jbr exposes a reusable helper component for defining Docker resource constraints.

For example, the following experiment is configured to use at most 90% of the CPU, and 10MB of memory.

{
  "@type": "SomeExperiment",
  "resourceConstraints": {
    "@type": "StaticDockerResourceConstraints",
    "cpu_percentage": 90,
    "memory_limit": "10m"
  }
}

All possible parameters (all are optional):

  • cpu_percentage: Percentage (0-100) of the total CPU power that can be used. E.g. when fully consuming 4 cores, this value must be set to 100.
  • memory_limit: Memory usage limit, e.g. '10m', '1g'.

Running against a different Docker instance

By default, Docker-based experiment will look for and use the Docker installation on your local machine. In some cases, you may want to run experiments within remote Docker instances. In such cases, you can use the -d or --dockerOptions option to pass a custom Docker options file.

For example, Docker options can be set when running an experiment as follows:

$ jbr run -d docker-options.json

docker-options.json for the default socket:

{
    "socketPath": "/var/run/docker.sock"
}

docker-options.json for running against a different host:

{
    "host": "http://192.168.1.10",
    "port": 3000
}

More configuration options can be found at https://github.com/apocas/dockerode#getting-started

License

jbr.js is written by Ruben Taelman.

This code is copyrighted by Ghent University – imec and released under the MIT license.