emp is a command line tool that helps you run and replicate experiments using the Empirical Framework

Dependencies

There are only 2 requirements:

• Node.js is a JavaScript runtime required to run this program on your computer. Download and install Node.js from here.
• Docker allows us to create self-contained portable environments that work accross different platforms. Follow these instructions to install Docker.

Install

emp is supported on Windows, Mac and Linux. Once you have node.js installed on your computer, you can install emp using npm.

npm install -g empirical-cli

Get started

Test your installation by replicating an experiment.

emp run empirical/mnist-sample/x/Hke43sz5e

This will clone the source code from GitHub, checkout the appropriate version, build the experiment image, download the required datasets, run the experiment, and save the results on your computer. All in one single command.

Run your experiments

Once you have defined your experiment using the Empirical Framework.

From GitHub

Run an experiment directly from a GitHub repository.

emp run mnist https://github.com/empiricalci/mnist-sample

Or provide a specific commit.

emp run mnist https://github.com/empiricalci/mnist-sample#0e6b04363c374992eb94a80dd2db0895711fb60f

From empiricalci.com

You can easily replicate any experiment pushed to empiricalci by running:

emp run <experimentId>

From a local directory

Or you can run an experiment from a local directory.

emp run my-protocol /path/to/code

Keep track of your experiments

Authenticate with the server

Authenticating with empiricalci.com allows you to save the results of your experiments and share them with your peers.

1. If you haven't done so, sign up for an account on empiricalci.com
2. Login using the CLI: emp login will ask for your credentials and store them on your computer

Push your results

Push your results to the online dashboard by providing a message a path to the output directory created by emp run and the project id.

emp push /path/to/report <projectId>

Configure

Empirical uses a directory to cache all the datasets downloaded and to save any files generated during the tests or experiments. This defaults to ~/empirical. You can change this by doing:

emp configure

Notes

GPU Support

GPU support is provided for Linux via nvidia-docker, you can follow the instructions on the "Quick start" to install it and learn how to run GPU enabled experiments here.

Currently Windows and MacOS are not supported since they rely on VMs to run Docker. See this issue for more information.

About Docker on Windows

If you're using the Docker for Windows version, as opposed to Docker toolbox make sure of the following:

• emp requires to mount certain directories on your containers, so make sure to share your local drives.
• emp can usually communicate with Docker using the default settings. If it fails to connect, make sure the DOCKER_HOST environment variable is set to the correct value. On Docker for Windows this defaults to 127.0.0.1:2375. You can set it using CMD as setx DOCKER_HOST "127.0.0.1:2375".