redibox-hook-trend
v1.2.3
Published
A scalable trending hook designed to track temporal trends in non-stationary categorical distributions.
Downloads
13
Maintainers
Readme
RediBox Trending Hook
This is a scalable trending hook designed to track temporal trends in non-stationary categorical distributions. It uses forget-table style data structures which decay observations over time. Using a ratio of two such sets decaying over different lifetimes, it picks up on changes to recent dynamics in your observations, whilst forgetting historical data responsibly. The technique is closely related to exponential moving average (EMA) ratios used for detecting trends in financial data.
Trends are encapsulated by a construct named Delta. A Delta consists of two sets of counters, each of which implements exponential time decay of the form:
Where the inverse of the decay rate (lambda) is the mean lifetime of an observation in the set. By normalising such a set by a set with half the decay rate, we obtain a trending score for each category in a distribution. This score expresses the change in the rate of observations of a category over the lifetime of the set, as a proportion in the range 0..1.
This implementation removes the need for manually sliding time windows or explicitly maintaining rolling counts, as observations naturally decay away over time. It's designed for heavy writes and sparse reads, as it implements decay at read time.
Each set is implemented as a redis sorted set
, and keys are scrubbed when a count is decayed to near zero, providing storage efficiency.
This hook handles distributions with upto around 106 active categories, receiving hundreds of writes per second, without much fuss. Its scalability is dependent on your redis deployment.
Installation
This is a hook for RediBox so you'll need that first. Then it's as simple as:
npm i redibox-hook-trend --save
RediBox will automatically load the hook for you.
Usage
The examples below assume Trend
as a reference to RediBox.hooks.trend
We also use trend
as a easier term for distribution
and item
instead of bin
.
Create a new trend (distribution)
Trend.create({
// the name of this trend
name: 'kittens',
// life time of this trend
time: 1209600, // 14 days
}).then(kittensTrend => {
// do things with your new trend
});
Check if a trend exists
Trend.exists('kittens').then(bool => {
// bool is truthy if trend exists
});
Get an existing trend or create a new trend
Trend.getOrCreate({
// the name of this trend
name: 'kittens',
// life time of this trend
time: 1209600, // 14 days
}).then(kittensTrend => {
// do things with the trend
});
Incrementing a trend item (bin)
kittensTrend.incr({
item: 'fluffball', // can even be json if needed
by: 1,
}).then(() => {
// all done
});
Fetch all items in a trend
These are pre-sorted highest score descending.
kittensTrend.fetch().then((items) => {
// console.dir(items);
});
Fetch a specific item in a trend
kittensTrend.fetch({
item: 'fluffball'
}).then((items) => {
// console.dir(items);
// [ { item: 'fluffball', score: 0.9999999999882863 } ]
});
Fetch a specific number of items in a trend
E.g. top 5 kittens in kittensTrend
kittensTrend.fetch({
limit: 5
,}).then((items) => {
// console.dir(items);
// [ { item: 'fluffball', score: 0.9999999999882863 } ]
// ...
// ...
});
Contributing
Full contributing guidelines are to be written, however please ensure you follow these points when sending in PRs:
- Ensure no lint warnings occur via
npm run lint
. - Implement tests for new features / functionality.
- Ensure coverage remains above 90% and does not decrease.
- Use verbose logging throughout for ease of debugging issues, see core.js for example.
- New modules should follow the same format as the default modules / template.
Note: For debugging purposes you may want to enable verbose logging via the config:
new RediBox({
log: {
level: 'verbose'
}
});
License
MIT