@fightmegg/scientist
v1.0.2
Published
A JavaScript library for carefully refactoring critical paths
Downloads
3
Maintainers
Readme
Scientist
A JavaScript library for carefully refactoring critical paths, port of GitHub Scientist
Installation
npm install @fightmegg/scientist
Usage
import Scientist from '@fightmegg/scientist'
const experiment = new Scientist('experiment #1');
experiment.use(() => 12); // old way
experiment.try(() => 10 - 8); // new way
experiment.publish(results => console.log(results)); // publish
const result = experiment.run();
// Sometimes we need to go Async
const experiment = new Scientist('experiment #1', { async: true });
experiment.use(async () => 12); // old way (async or promise)
experiment.try(async () => 10 - 8); // new way (async or promise)
experiment.publish(results => console.log(results)); // publish
const result = await experiment.run();
How to Science
Let's pretend you are changing how the result of a calculation is obtained. Tests can help you refactor, but really you want to compare the current and refactored behaviours under load.
Pass your original code to the use
function, and pass your new code / behaviour to the try
function. experiment.run
will always return whatever the use
function returns, but it does some stuff behind the scenes:
- It decided whether or not to run the try block
- Randomizes the order in which
use
andtry
functions are run - Measures the duration of all behaviours
- Compares the result of
try
and the result ofuse
- Swallow and record errors thrown in the
try
block - Publishes all of this information
The use
function is called the control. The try
function is called the candidate
If you do not declare a candidate, then the control is always ran and returned.
Creating useful experiments
The examples above are rather basic, the try
blocks don't run yet, and none of the results get published, lets improve upon that:
const experiment = new Scientist('experiment #1', { async: true });
experiment.enabled(() => {
// see "Ramping up experiments" below
return true;
})
experiment.publish(results => {
// see "Publishing results" below
console.log(results)
});
experiment.use(async () => 12);
experiment.try(async () => 10 - 8);
const result = await experiment.run();
Controlling comparison
Scientist compares control and candidate values using lodash.isequal
. To override this behaviour, use compare
to define how to compared observed values instead:
const experiment = new Scientist('experiment #1', { async: true });
experiment.use(async () => 12);
experiment.try(async () => 10 - 8);
experiment.compare((control, candidate) => {
return control.id === candidate.id
})
const result = await experiment.run();
Keeping it clean
Sometimes you dont want to store the full value for later analysis. For example an experiment may return a whole object instance, but you only care about a specific property. You can define how to clean these values in an experiment:
const experiment = new Scientist('experiment #1', { async: true });
experiment.use(async () => 12);
experiment.try(async () => 10 - 8);
experiment.clean((value) => {
return value.map(n => n.id).sort()
})
const result = await experiment.run();
And this cleaned value is available in observations in the final published result:
const experiment = new Scientist('experiment #1', { async: true });
// ...
experiment.publish((result) => {
console.log(result.control.value) // [<User Bob>, <User alice>]
console.log(result.control.cleaned_value) // [alice, Bob]
})
Ramping up experiments
As a scientist, you know its always important to be able to turn your experiment off, lest it run wild. In order to control whether or not an experiment is enabled, you must include the enabled
method in your implementation:
let percentEnabled = 100;
const experiment = new Scientist('experiment #1', { async: true });
// ...
experiment.enabled(() => {
return percentEnabled > 0 && (Math.random() * 100) < percentEnabled
})
const result = await experiment.run();
This code will be invoked for every method with an experiment every time, so be sensitive about its performance
Publishing results
Whats the point of being a Scientist if you can't publish your results?
You must implement the publish(result)
method, and you can publish the data however you like. For example timing data could be sent to graphite, and mismatches coule be sent to a debugging service.
The structure of result
is:
{
"name": "<Experiment Name>",
"matched": true,
"execution_order": ["control", "candidate"],
"error": false,
"control": {
"duration": 100,
"value": 20,
"cleaned_value": 20,
"startTime": 100002320,
"endTime": 1000021000,
"error": "error ...",
},
"candidate": {
"duration": 100,
"value": 20,
"cleaned_value": 20,
"startTime": 100002320,
"endTime": 1000021000,
"error": "error ...",
},
}
Handling errors
Scientist handles and tracks all errors raised in a try
or use
block. Scientist also calls publish
which contains all of the information about any errors.
In control code
If an error is thrown in a use
block, it will eventually be thrown by the Scientist module (in order to mimick real world), therefore you will want to handle this properly.
In candidate code
If an error is thrown in a try
block, it will be swallowed and not thrown, you will want to handle this in your publish
block.
Breaking the rules
Sometimes it can be useful to break the rules.
Run Control or Candidate only
There will be times when you cannot have both the control and candidate running sequentially (such as performing a database update, but l would not advise using this library for that).
In order to get around that, we can use the run_only
method, this will either run the candidate or control, and return the result from whatever function you ran:
const experiment = new Scientist('experiment #1', { async: true });
experiment.use(async () => 12);
experiment.try(async () => 10 - 8);
const candidateResult = await experiment.run_only('candidate');
// OR
const controlResult = await experiment.run_only('control');
In both scenarios above, the publish
method will still be invovked, but the control
or candidate
properties will be null
depending on which one you ran.
A/B Testing
Similar to the above, we can utilize the run_only
method to perform A/B testing:
const experiment = new Scientist('experiment #1', { async: true });
experiment.use(async () => 12);
experiment.try(async () => 10 - 8);
const result = experiment.run_only(['control', 'candidate'][Math.round(Math.random())]);