testpilot-metrics

The testpilot-metrics library sends pings to Mozilla's internal metrics pipeline and, optionally, to Google Analytics. It is designed for use by Test Pilot experiments, and should be used in privileged Firefox extension code--not in regular web pages.

Installation

npm install testpilot-metrics

The only file you need in your build chain is testpilot-metrics.js. It has no dependencies.

Google Analytics setup

1. Create a GA account, if you don't have one.

2. Create a mobile app GA property for your experiment.

• The app type has version number tracking baked in, which helps with connecting releases to changes in usage or error rates over time.

• The app dashboard also has a real-time event view, which you can use to watch test events land while debugging.

3. If you think you might want to take advantage of Test Pilot's A/B testing support, then create a Custom Dimension that you can use to track variant information. By convention, the first custom dimension, cd1, should be used by testpilot-metrics for recording variant info. You can use the higher values for other experiment-specific extra fields you want to send to GA.

Usage

Quick start

Here's a simple node-style example:

const Metrics = require('testpilot-metrics');

const { sendEvent } = new Metrics({
  id: '@my-addon',
  version: '1.0.2a',

  // These keys are also required if you're using GA:
  uid: 'some-non-PII-user-ID',
  tid: 'UA-XXXXXXXX-YY'
});

// Sometime later, when a button gets clicked:
sendEvent({
  object: 'toolbar-button',
  method: 'click'
});

Generating a uid to send to GA

There are many ways to generate a unique, non-PII user ID. Here are some examples:

WebExtension:

// returns a cryptographically strong random integer
// example: 3696675532
const uid = window.crypto.getRandomValues(new Uint32Array(1))[0];

SDK:

const { uuid } = require('sdk/util/uuid');

// returns a random uuid v4 string
// example: "b0aef941-35c0-6748-912f-d779c6e4d05e"
const uid = uuid().number.replace('{','').replace('}','');

Sending extra fields in addition to method / object / category

If you need to send fields in addition to the defaults, you'll need to follow different steps for GA and for Mozilla's data pipeline.

Mozilla support

For Mozilla's data pipeline, you'll need to define a Redshift schema that includes all parameters, their types, and their size. See the Test Pilot metrics docs for more details.

Once you've defined the storage schema, you can simply insert extra fields as top-level keys in the sendEvent parameter object, for example:

// Track the `clientX` and `clientY` values of an experiment popup window,
// and send along with the method, object, and category:

sendEvent({
  object: 'special-button',
  method: 'click',
  category: 'popup',
  clientX: 185,
  clientY: 560
});

Sending custom fields to Google Analytics requires one additional step.

Advanced Google Analytics support: the transform function parameter

Google Analytics doesn't support arbitrary named parameters. To send extra fields to GA, you must decide how to map your extra fields to the GA custom fields.

Once you've figured out this mapping, pass a second argument to sendEvent: a transform function that will convert the extra fields to GA custom fields.

The transform function is called with 2 arguments: first, the raw event object that was passed to sendEvent; second, the default GA ping that would normally be submitted (ignoring any extra parameters). The transform function should return a JS object whose keys are the custom set of GA parameters. The testpilot-metrics library will then encode and send the transform function's output.

Note that the first custom dimension, cd1, is reserved for variant testing. So, taking the example from the last section, you might map the extra clientX field to cd2, and clientY to cd3, and otherwise leave the GA event object alone. You'd do this:

sendEvent({
  object: 'special-button',
  method: 'click',
  category: 'popup',
  clientX: 185,
  clientY: 560
},
transform: (input, output) => {
  // Add two extra fields from the input object to the output object.
  output.cd2 = input.clientX;
  output.cd3 = input.clientY;
  // Return the transformed output object, to be encoded and sent to GA.
  return output;
});

Google Analytics defines 8 different hit types, but this library only uses the Event hit type by default. Transform functions allow experiment authors to submit other hit types. For example, uncaught Errors could be sent using the GA exception type:

try {
  somethingImportant();
} catch (err) {
  console.error(`somethingImportant failed: ${ex}`);
  sendEvent({
    // This (method, object) pair will be sent to Mozilla's internal metrics.
    method: 'uncaught-exception',
    object: 'somethingImportant'
  },
  transform: (input, output) => {
    // Change the GA hit type.
    output.t = 'exception';
    // Add the exception description field, mandatory for 'exception' hits.
    output.exd = `somethingImportant: ${err}`
    return output;
  });
}

API docs

See API.md.

WebExtension Usage

Follow the steps below to add testpilot-metrics to your WebExtension. You can also look at the example WebExtension in this repo for a more in-depth example.

1. Add the testpilot-metrics.js file to your project :-)

2. Add the GA URL permission to your manifest.json:

// manifest.json
{ ...
  "permissions": [ "https://ssl.google-analytics.com/collect" ],
  ...
}

3. Add the testpilot-metrics.js file to the list of background scripts in manifest.json, before the background script that will use the library:

// manifest.json
{ ...
  "background": {
    "scripts": [ "testpilot-metrics.js", "background.js" ]
  },
  ...
}

4. In your startup code, call the Metrics constructor, passing in your add-on's ID (id) and version (version), and, if you are using Google Analytics, a non-PII user ID (uid) and your Google Analytics tracking ID (tid).

// background.js startup
const { sendEvent } = new Metrics({
  id: '[email protected]',
  version: '0.0.1',
  tid: 'UA-XXXXXXXX-YY',
  uid: '123-456-7890' // this can be any non-PII identifier that is stable over time
});

5. Each time an interesting event occurs, call sendEvent, passing in event details in method/object/category format. If you are running any multivariate or A/B tests, you can include that info as well:

// Example click handler, somewhere in background.js
browser.browserAction.onClicked.addListener((evt) => {
  sendEvent({
    object: 'webext-button',
    method: 'click',

    // these fields are optional:
    category: 'toolbar',
    variant: 'green-button'
  });
});

SDK Usage

Follow the steps below to add testpilot-metrics to your SDK add-on. You can also look at the example SDK add-on in this repo for a more in-depth example.

1. Add the testpilot-metrics.js file to your project :-)

2. Load the library using the SDK loader:

const Metrics = require('testpilot-metrics');

3. In your startup code, call the Metrics constructor, passing in your add-on's ID (id) and version (version), the type argument with value sdk, and, if you are using Google Analytics, a non-PII user ID (uid), and your Google Analytics tracking ID (tid):

const { sendEvent } = new Metrics({
  id: '[email protected]',
  version: '1.0.2',
  type: 'sdk',
  tid: 'UA-XXXXXXXX-YY',
  uid: '123-456-7890' // this can be any non-PII identifier that is stable over time
});

4. Each time an interesting event occurs, call sendEvent, passing in event details in method/object/category format. If you are running any multivariate or A/B tests, you can include that info as well:

const btn = ui.ActionButton({
  id: 'metrics-test-button',
  label: 'Metrics Test Button',
  icon: {
    '16': './icon-16.png',
    '32': './icon-32.png',
    '64': './icon-64.png'
  },
  onClick: () => {
    sendEvent({
      object: 'sdk-button',
      method: 'click',

      // these fields are optional:
      category: 'interactions',
      variant: 'green-button'
    });
  }
});

Google Analytics output

For simplicity, the only GA hit type currently supported is the Event hit type.

As an example, the following ping:

const { sendEvent } = new Metrics({
  id: '@my-addon',
  version: '1.2.3',
  tid: 'UA-XXXXXXXX-YY',
  uid: '110ec58a-a0f2-4ac4-8393-c866d813b8d1'
});

sendEvent({
  method: 'click',
  object: 'home-button-1',
  category: 'toolbar-menu',
  variant: 'green-button'
});

is transformed into a GA Measurement Protocol hit with parameters:

msg = {
     v: 1,
    an: '@my-addon',
    av: '1.2.3',
   tid: 'UA-XXXXXXXX-YY',
   // NOTE: the 'uid' parameter is mapped to the GA 'cid' parameter, see issue #28
   cid: '110ec58a-a0f2-4ac4-8393-c866d813b8d1'
     t: 'event',
    ec: 'toolbar-menu',
    ea: 'click',
    el: 'home-button-1',
   cd1: 'green-button'
}

These parameters are URL encoded before sending.

Interested in contributing?

Great! Grab a bug and/or say hello in the testpilot channel on Mozilla IRC :-)

Code of Conduct

Note that all contributors are expected to be cool.

Pull Request checklist

Make sure you do this stuff before opening your PR / look at this stuff when reviewing a PR:

1. update code docs, if needed

2. rebuild API docs: npm run build-docs and add changes to the PR

3. make sure tests are green: npm run test and add new tests if you add code

4. update the README and examples to document any user-visible changes you've made

License

MPL 2.0

Author

Brought to you by @6a68 and the Test Pilot team at Mozilla.