Benchmark

Benchmark is a simple benchmarking tool for GPU.js. This tool works both in JavaScript and CLI. This tool runs three benchmarks:

1. Matrix Multiplication
2. Matrix Convolution
3. Pipelining

Table of Contents

• Installation
• Browser Usage
• Usage
• CLI Usage
• Options
• Saving Graphs As JSON
• Multiple Benchmarks
• Output
• Stats
• BenchmarkOut Object
• Benchmarks
• Using With React Native

Installation

NOTE: The package gpu.js needs to be installed separately. Benchmark is available on npm under the name @gpujs/benchmark.

Using yarn

yarn add @gpujs/benchmark

Using npm

npm install @gpujs/benchmark

NOTE: If it asks for a GPU.js version, you can choose any version of your choice (>=v2.0.0) but the provided dist files will have the version which was the latest during the release of that version of benchmark.

Browser Usage

Building

NOTE: The latest dist files are not included since v2.1.0 due to problems with browserify(https://github.com/gpujs/benchmark/issues/7). This issue will be addressed as soon as possible. NOTE: The dist files are also included in the npm module and GitHub repository, skip this step if you are not running a modified script locally. We use browserify and minify to build the distributable files dist/benchmark.js and dist/benchmark.min.js. After running the setup script, run the following command

yarn build

Using

Include the benchmark dist file in the HTML file.

<script src="path/to/dist/gpu.min.js"></script> <!--gpu.js has tp be included separately-->
<script src="path/to/dist/benchmark.min.js"></script>

or, from the npm module

<script src="path/to/dist/gpu.min.js"></script> <!--gpu.js has tp be included separately-->
<script src="path/to/node_modules/@gpujs/benchmark/dist/benchmark.min.js"></script>

The exported function is benchmark.

const out = benchmark(options)

NOTE: Options are is an Object. See this.

Usage

Javascript

1. Import Benchmark.

const benchmark = require('@gpujs/benchmark')

OR using ES6 syntax

import benchmark from '@gpujs/benchmark'

2. Run it.

const benchmarks = benchmark.benchmark(options)

OR run Multiple Benchmarks

const benchmarks = benchmark.multipleBenchmark(options)

This returns the benchmarks in an Object. See this.

NOTE: Options are is an Object. See this.

CLI

1. Clone the repository and open the directory.

git clone https://github.com/gpujs/benchmark
cd benchmark

2. Install yarn We use yarn as our package manager. You will have to install that too, as a side effect. (If you have yarn installed, skip this step)

npm install -g yarn

3. Install the dependencies

yarn setup

NOTE: If it asks for a GPU.js version, you can choose any version of your choice (>=v2.0.0) but the provided dist files will have the version which was the latest during the release of the latest version of benchmark

4. Run the tool in the CLI

yarn start

OR using node

node ./index.js

This will prompt you to enter the optional [options]

Using CLI with JSON Options as Input

yarn start options

options is a stringified JSON object passed as an argument.

OR using node

node ./index.js options

Here, options is a stringified JSON object.

Example

yarn start '{"num_iterations": 4}'

Options

The following options can be passed on to the benchmark or multipleBenchmark method.

1. benchmark options:

• cpu(Object) *: A custom GPU({mode: 'cpu'}) Object to benchmark specific versions of GPU.js(>= v2.0.0). Mandatory in everything except CLI.

• gpu(Object) *: A custom GPU() Object to benchmark specific versions of GPU.js(>= v2.0.0). (default: The version shipped with benchmark). Mandatory in everything except CLI.

• matrix_size(Integer): The size of the uniform matrix used for benchmarking. (default: 512)

• num_iterations(Integer): The number of iterations of run time calculation. (default: 1)

• logs(Boolean): Toggles console logs by the library.

• cpu_benchmark(Boolean): Toggles the benchmarking of CPU. False is recommended to big matrix sizes. (default: true)

2. multipleBenchmark options: Multiple Benchmark options have the following structure.

{
  common_options: { // options common to all but can be overridden in range or in full_options, preference given to range
    cpu_benchmark: false,
    cpu: new CPU({mode: 'cpu'}),
    gpu: new GPU()
  },
  range: { // only one of this and full_options works
    option_name: 'matrix_size',
    interval: [128, 1024],
    step: 100 //(default 10)(A.P.: 128, 138, 148, 158) one of step or common_ratio can be used, preference given to step
    // common_ratio: 2 (G.P.: 128, 256, 512, 1024)
  },
  full_options: [
    {
      // array of options objects for each benchmark(only one of this and range works, preference given to range)
    }
  ]
}

• common_options(Object): Options common to all the benchmarks that are run. (Same as benchmark options).
• range(Object): Used to create a set of options using a set of rules, for each benchmark. (only one of range or full_options can be used)
  • option_name(String): The option for which the range is applied. This has to be of type Number. It can be one of the benchmark options.
  • interval(Array): The upper and lower limits for the option.
  • step(Number): The common difference between each option value. All the options will be in an AP. (only one of step or common_ratio can be used, preference is given to step)
  • common_ratio(Number): The common ratio between each option value. All the options will be in a GP. (only one of step or common_ratio can be used, preference is given to step)
• full_options(Array): An array of options object, each one corresponding to one benchmark. Each object is the same as benchmark options. (only one of range or full_options can be used)

Multiple Benchmarks in CLI

yarn start --multiple [options?]

options to the CLI are stored in a stringified JSON object passed as an argument. More about Multiple Benchmarks.

Saving Graphs as JSON

1. Plotly Style JSON

yarn start --multiple --returnPlotlyJSON

This will log to the console, plotly.js style JSON which stores the graph data for GPU score v/s matrix size of each benchmark.

yarn start --multiple --savePlotlyJSONToFile=path/to/file.json

This saves the plotly.js style JSON data for:

• GPU score v/s matrix size
• GPU matrix multiplication run time v/s matrix size
• CPU score v/s matrix size
• CPU matrix multiplication run time v/s matrix size

NOTE: If CPU is not benchmarked, CPU score and run time will have non-meaningful negative values which are to be ignored.

NOTE: Filename need not have a .json extension.

1. Chartist Style JSON

yarn start --multiple --returnChartistJSON

This will log to the console, chartist.js style JSON which stores the graph data for GPU score v/s matrix size of each benchmark.

yarn start --multiple --saveChartistJSONToFile=path/to/file.json

This saves the chartist.js style JSON data for:

• GPU score v/s matrix size
• GPU matrix multiplication run time v/s matrix size
• CPU score v/s matrix size
• CPU matrix multiplication run time v/s matrix size

NOTE: If CPU is not benchmarked, CPU score and run time will have non-meaningful negative values which are to be ignored.

NOTE: Filename need not have a .json extension.

NOTE: One or more of the above arguments for JSON output can be used with --multiple

Multiple Benchmarks

Benchmark allows you to run a sequence of benchmarks each with different custom options or each having number options like matrix size changed by a fixed amount.

benchmark.multipleBenchmark(options);

Where options is an object with the following properties:

• common_options(Object): Options common to every benchmark in a sequence. (default: {cpu_benchmark: false})
• range(Object): Define a range of option(number type) values, one for each benchmark in the sequence. e.g.: matrix_size: 512, 1024, 1536... or matrix_size: 512, 1024, 2048 ... Here, the specified option can either be incremented by a fixed number(common difference) or multiplied by a fixed number(common factor).
  • option_name(String): The name of the option for which the range is to be set. e.g.: matrix_size (Default: matrix_size)
  • interval(Array): An array with upper and lower limits for the range. e.g.: [512, 2048] (Default: [128, 1024])
  • step(Number): The fixed number which is to be added(common difference). (Default: 100)
  • common_ratio(Number): The fixed number to be multiplied. (Default: none)

NOTE: Only one of step and common_ratio can be used

• full_options(Array): An array of objects specifying separate set of options for each benchmark in the sequence(common_options properties can be overridden here). (Default: none)

NOTE: Only one of range and full_options can be used

Examples

1. Range:

benchmark.multipleBenchmark({
  common_options: {
    cpu_benchmark: false,
    logs: false
  },
  range: {
    option_name: 'matrix_size',
    interval: [128, 2048],
    common_ratio: 2
  }
})

The above code runs a separate benchmark for the matrix sizes 128, 256, 512, 1024, 2048 which are in GP.

2. full_options:

benchmark.multipleBenchmark({
  common_options: {
    logs: false,
    cpu_benchmark: false
  },
  full_options: [
    {
      logs: true, // override
      matrix_size: 2048
    },
    {
      cpu_benchmark: true, //override
      matrix_size: 128
    }
  ]
})

API

Output

The output of any benchmark(multiple or single) is a BenchmarkOut Object.

Stats

The output contains a stats property which shows the overall stats of the benchmark:

• run_time: The run time stats

  • mat_mult, mat_conv, pipe(Object): These three objects contain the stats for each type of benchmark.
    • diff: Has a single property that contains performance comparison scores between CPU and GPU.
      • cpu_gpu:
        • min, max, avg: The minimum, maximum and average time taken stats
          • winner(gpu | cpu): The better performer among the two.
          • percentage(Number): By how much percentage it is better.

• build_time: The build time stats

  • mat_mult, mat_conv: Built time stats for each benchmark.
    • diff: Same as the diff object in run_time except that it compares GPU v/s GPU(pipeline mode) in the property gpu_pipe. (P.S. Best Performer and Worst Performer are not included)

• overall: The overall stats mat_mult, mat_conv: Overall stats for each benchmark

  • best_performer(gpu | cpu): The best overall performer.
  • worst_performer(gpu | cpu): The worst overall performer.
  • diff: Same as the diff object in run_time

• score: The score object is a property of the main output object.

  • gpu, cpu(Number): A score is a number representing the overall normalized average performance of the GPU or CPU. This score can be directly compared to other benchmarks or hardware.

TECHNICAL: The score is floor of one-hundredth of the ratio of the total number of operations in matrix multiplication to the time taken for the operations.

• In the case of matrix multiplication, one single operation is taken to be the product of two array elements and the total number of operations is taken to be the cube of one of the dimensions[for a square matrix].

BenchmarkOut

This object stores the output of Benchmark.

Properties

• mat_gen, mat_pad(Number): Matrix generation and matrix padding times in ms.
• build_time(Object):
  • mat_mult, mat_conv(Object)
    • gpu, pipe(Number): Compile times for GPU and GPU(pipeline mode) in ms for each benchmark.
• run_time(Object): Run times for each benchmark.
  • mat_mult, mat_conv, pipe(Object): Run times for each benchmark.
    • gpu, cpu(Object): GPU and CPU run times.
      • min, max, avg(Number): The minimum, maximum and average run times in ms.
      • deviation (Number): Percentage deviation of results from average value.
• stats(Object): The statistics.

Methods

• getDataField(field, index = 0)(returns: ***): Gets any one of the output field(property).
  • field(String): The name of the field.
  • index(Number): The index of the benchmark if multiple benchmarks are run.
• getPlotlyJSON(compare_fields), getChartistJSON(compare_fields)(Returns: Array): Returns plotly or Chartist style JSON Object for charts. (only for multiple benchmarks)
  • compare_fields: An array of objects having two properties x and y representing the data to be plotted on their respective axes.
    • x, y(String): Can be one of:
      • matrix_size
      • gpu_score
      • cpu_score
      • gpu_run_time_mat_mult: GPU matrix multiplication run time
      • cpu_run_time_mat_mult: CPU matrix multiplication run time
      • gpu_run_time_mat_conv: GPU matrix convolution run time
      • cpu_run_time_mat_conv: CPU matrix convolution run time
      • pipe_run_time: GPU pipelining run time

Default value of compare_fields argument for getPlotlyJSON and getChartistJSON methods:

[
  {
    x: 'matrix_size',
    y: 'gpu_run_time_mat_mult'
  },
  {
    x: 'matrix_size',
    y: 'pipe_run_time'
  },
  {
    x: 'matrix_size',
    y: 'gpu_score'
  }
]

Benchmarks

Matrix Multiplication

This benchmark multiplies two randomly generated uniform-sized matrices and benchmarks the GPU and CPU against the time taken by each.

GPU.js Kernel:

function(a, b) {
  let sum = 0;
  for (let i = 0; i < this.output.x; i++) {
    sum += a[this.thread.y][i] * b[i][this.thread.x];
  }
  return sum;
}

Matrix Convolution

This benchmark convolves a 3x3 kernel over a randomly generated uniform sized matrix. The convolution kernel is

1 2 1
2 1 2
1 2 1

GPU.js Kernel:

function (array, kernel) {
  let sum = 0;
  for (let i = 0; i < ${kernelX}; i++){
    for (let j = 0; j < ${kernelY}; j++){
      sum += kernel[j][i] * array[this.thread.y + j][this.thread.x + i];
    }
  }
  return sum;
}

Where kernelX and kernelY are the dimensions of the kernel.

Pipelining

GPU.js supports a feature called Pipelining and this benchmark benchmarks this feature. It runs four matrix multiplication benchmarks in a sequence while pipelining the output of the earlier benchmark to be used as an input to the next one. The benchmark is run both on the GPU and the CPU(without pipelining) and the time taken is compared. When it is run on the GPU, the output of the previous multiplication is passed on to the next call as a texture (a storage unit on the GPU) on the GPU itself which drastically reduces the time taken because the output need not be converted and transferred to the CPU and back.

Expo

GPU.js can be run on Android and iOS devices using expo-gl which is a simple package developed by the GPU.js community.