Ember Experiments

A split testing framework for easily adding multivariate or A/B testing to Ember applications

Installation

• ember install ember-experiments

Defining Experiments

Basic A/B Experiments

A/B testing allows you to split your audience into 2 test groups

  // app/routes/application.js
  import Route from '@ember/routing/route';
  import { inject as service } from '@ember/service';

  export default Route.extend({
    experiments: service(),

    setupController(controller, model) {
      this._super(controller, model);

      this.get('experiments').setup('experimentName', {
        a: 50, // the provided int determines what percentage of users should receive this variation
        b: 50
      });
    }
  });

In the above example, 50% of users will receive variant a and 50% will receive variant b

Accessing variations in js

  //app/components/my-component.js
  import Component from '@ember/component';
  import { inject as service } from '@ember/service';

  export default Component.extend({
    experiments: service(),
    didInsertElement() {
      console.log('current variation', this.get('experiments').getVariation('experimentName'));
      console.log('are we in variation a?', this.get('experiments').isEnabled('experimentName', 'a'));
    }

Accessing variations in templates

You also have access to experiment selections by concatting them together as camelCase. For example, if your experiment is named user test 1 and your variations are a and b, you could access them in templates as:

  {{#if (experiment 'userTest1' 'a')}}
    Here we are in variation A
  {{else if (experiment 'userTest1' 'b')}}
    Here we are in variation B
  {{/if}}

You also have access to computed vars representing each test variation:

  {{#if experiments.userTest1A}}
    Here we are in variation A
  {{else if experiments.userTest1B}}
    Here we are in variation B
  {{/if}}

Multivariate Testing

In addition to traditional A/B testing, you can also specify multivariate tests (A/B/C/D)

  import Route from '@ember/routing/route';
  import { inject as service } from '@ember/service';

  export default Route.extend({
    experiments: service(),

    setupController(controller, model) {
      this._super(controller, model);

      this.get('experiments').setup('experimentName', {
        a: 10,
        b: 50,
        c: 40
      });
    }
  });

In the above example, 10% of users will get variant a, 50% will get variant b and 40% will get variant c

Working with promises or route hooks

The setup method returns a promise, allowing you to delay rendering until a variant is picked if needed

  import Route from '@ember/routing/route';
  import { inject as service } from '@ember/service';

  export default Route.extend({
    experiments: service(),

    model() {
      return this.get('experiments').setup('experimentName', {
        a: 50,
        b: 50
      }).then(variation => {
        switch(variation) {
          case 'a':
            // do the things we want to do with variation a
            break;
          case 'b':
            // do the things we want to do with variation b
            break;
        }
      });
    }
  });

Fetching test variations server-side

Available in v1.1.0-beta.1

If you have a server-side service that you need to fetch experiments from, instead of setting them up client-side, the experiments service has two properties that you can optionally set and later access.

Ember Concurrency task

You can set the serverFetchTask property to be an Ember Concurrency task that you define and perform. This task should set the API response to the experiments service:

// application route
experiments: service(),

beforeModel() {
  const fetchExperimentsTask = this.fetchExperiments;
  // don't wait for this, unless you want to block rendering
  fetchExperimentsTask.perform();
  this.experiments.set('serverFetchTask', fetchExperimentsTask);
},

fetchExperiments: task(function * () {
  const response = yield fetch('/experiments');
  const { experiments } = yield response.json();
  // { experiment1: 'control', experiment2: 'enabled' }
  this.experiments.setExperiments(experiments);
  return;
})

Now you can access the task's state, and you can access the experiment values, all from the experiments service:

experiments: service()

// are we waiting for the experiments api response?
this.experiments.serverFetchTask.isRunning

// wait for the experiments api response
await this.experiments.serverFetchTask.last

// get an experiment variant
this.experiments.getVariation('experiment1')

Plain JavaScript Promise

While the use of an ember concurrency task as defined above is the recommended approach, you can also use a Promise and set it to the serverFetchPromise property. This promise should set the API response to the experiments service:

// application route
experiments: service(),

beforeModel() {
  // don't wait for this, unless you want to block rendering
  const fetchExperimentsPromise = fetch('/experiments').then((response) => {
    return response.json();
  });
  fetchExperimentsPromise.then((experiments) => {
    // { experiment1: 'control', experiment2: 'enabled' }
    this.experiments.setExperiments(experiments);
  });
  this.experiments.set('serverFetchPromise', fetchExperimentsPromise);
}

Now you can access the promise, and you can access the experiment values, all from the experiments service:

experiments: service()
...
this.experiments.serverFetchPromise.then(() => {
  this.experiments.getVariation('experiment1')
})

Manually enabling test variations

If you need to manually enable/disable any test variations, you can do so by pulling in the provided activate-experiments mixin into your Application route.

  import Route from '@ember/routing/route';
  import ActivateExeriments from 'ember-experiments/mixins/activate-experiments';

  export default Route.extend(ActivateExeriments, {
  });

Once that's done, you can then activate any experiment by adding ?experiments=expName/variantName to your URL. To activate multiple tests, add them using CSV ?experiments=expName/variantName,expName2/variantName2.

Using Ember-Experiments with Ember-Metrics

ember-metrics is a great library for adding various metric libraries to your app, such as Google Tag Manager or Mixpanel. Ember Experiments has been made to work easily with the library to add experiment data to event calls. To do so, you'll want to extend the ember-metrics service in your app

// app/services/metrics
import Metrics from 'ember-metrics/services/metrics';
import { inject as service } from '@ember/service';

export default Metrics.extend({

  experiments: service(),

  trackEvent(...args) {
    let eventData = args[args.length - 1];
    args[args.length - 1] = Object.assign({}, eventData, this.get('experiments').getExperiments());
    this._super(...args);
  }

});

Test Support

Adding a call to setupExperiments in your test module provides access to this.experiments, and also automatically cleans up experiments after each test.

import setupExperiments from 'ember-experiments/test-support/setup-experiments';

module('Acceptance | experiments', function(hooks) {
  setupApplicationTest(hooks);
  setupExperiments(hooks);

  test('experiments in testing me knees', async function(assert) {
    this.experiments.enable ('knee', 'left');
    await visit('/activate');

    let service = this.owner.lookup('service:experiments');

    assert.ok(this.experiments.isEnabled('knee', 'left'));
  });
}})

By passing an options hash with inTesting to your experiments.setup hash, your test suite will use that variation unless overridden by enabling another variation. In the example below, control will be the variation used in your test suite. You're able to override the inTesting variation by using this.experiments.enable('experimentName', 'variation')

setupController(controller, model) {
  this._super(controller, model);

  this.get('experiments').setup('experimentName', {
    control: 50,
    enabled: 50
  }, {
    inTesting: 'control'
  });
}

Good To Know's

• It's safe to set up the same test as many times as you'd like. The test is enabled and a variant is selected on the first setup. Subsequent setups will abort immediately and return the originally selected variant.
• Selected variants are stored in a cookie set to expire by default in 365 days. You can extend the exeriments service and set cookieName and cookieMaxAge to customize these values.

Experiments Service API

• setup('experimentName', variations = {}, options = {}) Allows you to set up an experiment for future use. variations is an object containing possible variations as keys, and the probability of hitting that variation as an int for the value. Returns a promise with the selected variant as the only parameter. options can include the isTesting value (see above)
• enable('experimentName', 'variantName') Force enable a variant for an experiment. The experiment does NOT need to be predefined, and you do NOT need to specify a variant that was passed into a setup call. You can force experiments to any variant name you'd like
• isEnabled('experimentName', 'variantName') Returns true/false based on if experimentName is currently set to variantName
• getVariation('experimentName') Returns the current variation for a provided experiment
• alreadyDefined('experimentName') Returns true/false based on if an experiment has already been set up
• getExperiments() Returns an object of experiments as keys with their current variant as values
• setExperiments({experimentName: variantName}) Allows you to force-set all experiments to specific values
• clearExperiments() Clear all experiments

Special Thanks

We've been A/B testing internally at Outdoorsy for quite some time now using a hodgepodge of internal tools on top of Ember Feature Flags. It served us well, but was time for some changes to make it easier to track experiments and perform more complicated split testing. Massive thanks to Katie for the awesome addon that got us started!

Contributing

Package uses Yarn for dependency management. If you're adding a new dependency, please make sure you run yarn before pushing back upstream.

Getting Started

• get clone this repository
• yarn in the created folder

PR Requirements

• All new functionality should be submitted with tests
• Please run yarn before opening a PR if you're adding a new dependency

Running tests

• ember test – Runs the test suite on the current Ember version
• ember test --server – Runs the test suite in "watch mode"
• ember try:each – Runs the test suite against multiple Ember versions

Running the dummy application

• ember serve
• Visit the dummy application at http://localhost:4200.

For more information on using ember-cli, visit https://ember-cli.com/.

License

This project is licensed under the MIT License.