VectorDataViews

Note: this README is outdated

VERSION: v0.0.44, build 20180625a

Abel Vázquez Juan Domínguez Javier Aragón

This tiny module eases the access and management of the data attached to vector tiles when rendered using MapboxGL JS, using a centralized dataview that provides all the needed functionality.

It also provides support for dictionary-encoded data (tree => sorted array).

Performance-wise: In the included demo, processing 1000 features x 700 KPIs takes less than 300ms. This processing includes: refresh the data in the dataview, filtering, styling and generating histogram per dictionary-encoded KPI

TO-DO

Requirements:

• [x] webworkers
• [x] destroy
• [x] aggs category options
• [x] get growth for color arrow in ranking items
• [x] manage styling with dictionary-encoded data
• [x] abstract all the settings to tiles.json metadata
• [x] manage filtering with dictionary-encoded data
• [x] improve decoding/rendering performance - Part I
• [x] improve internal performance - Part II
• [x] getSeries() for any parameter
• [ ] ...

Fixes

• [x] clean demo code
• [x] update demo with visual callback
• [x] limit/manage the data refresh rate
• [x] change the trigger event from renderto something less agressive
• [x] fix the autoinit
• [x] check mbtiles metadata returns: sourcelayername, max-min zooms,... etc.
• [x] fixed setKPI to adapt it to new code
• [x] tested against MapboxGL JS 0.45.0-beta1, workaround for their issue 6555
• [x] removed evil eval, improved performance of the involved statement 99.4% as a side effect (jsperf)
• [x] fixed ranking sorting
• [x] improved workers performance and avoid race conditions
• [x] fixed init NaN for KPI
• [x] fixed number of rendered features for zoom levels above 15.0
• [x] way more responsive mmoving processing to worker
• [x] regression in filtering functions
• [x] fixed nulls management
• [ ] review camelcase
• [ ] ...

Nice to have

• [x] improve decoding/rendering performance
• [ ] enhance the dictionary with human readable labels
• [ ] generate proper documentation
• [x] some sugar syntax to make the methods human usable
• [x] add more aggregates? median, percentiles, stddev,...?
• [x] add normalization using z-score in order to compare different areas?
• [ ] add JSdoc
• [ ] functional styles
• [ ] ...

Pre-Requisites for dictionary-encoded data

Dictionary-encoded data is just a compact way to deliver large tree-like data as a plain array, and a dictionary object that describes the tree.

In order to enable the dictionary-encoded data, the layer served as MVT should include an extra metadata property named dict like

"dict": {
    "year": [
        "2014",
        "2015",
        "2016",
        "2017",
        "2018"
    ],
    "item": [
        "total",
        "apples",
        "oranges",
        "lemons",
        "bananas",
        "watermellons"
    ],
    "kpi": [
        "kg_sold",
        "kg_rotten",
        "avg_price_kg",
        "avg_kg_per_sale",
        "number_of_sales",
        "number_of_claims",
        "number_of_pies",
        "growth",
        "stability",
        "the_answer_to_life_the_universe_and_everything"
    ]
}

So the data array for each feature is built starting with all the KPIs for year = 2014, item = total, then, all the KPIs for year = 2014, item = apples, ..., and ends with the KPIs for year = 2018, item = watermellons. And finally served as a plain string property (MVT standard does not allow array properties) of the feature.

Filters

To find an specific value in our array, a filter will be needed. These filters are defined aas objects with a value assigned to each branch of the tree, as a coordinate system

{year: '2015', item:'apple', kpi:'number_of_pies'}

Instantiate the VectorDataView

Once the map is loaded and our layer added, it's the moment to instantiate a new VectorDataView.

It needs to be fed with an object with the next properties:

• map: a reference to the MapboxGL JS map object.
• layername: the name / id of the layer.
• sourcename: the nameof the data source used by the layer
• dataproperty (optional): the name of a property that will store dictionary-encoded data.
• uniqueid: propertit that identifies unique features to group split pieces of the same feature. Vg. 'cartodb_id'.
• kpi: object that defines the initial KPI, shaped as `{property <, filter>}
• style: object that describes the style applied to the KPI based choropleth
  • breaks: breaks array
  • colors: colors array
  • zcolors: colors array for the z-score
• callback (optional): reference to a callback function that will be called each time the data is updated.
• filter: filter , to prefilter the features not in use yet
• filterNulls: boolean that set whether the features with null KPI will be used or not

Example:

    let options = {
        'map': map,
        'layername': 'blocks',
        'sourcename':  'mydatasource'
        'dataproperty': 'data',
        'uniqueids':'geoid',
        'kpi':{
            property: 'data',
            filter: {
                level0: "2018-02-01",
                level1: "total retail",
                level2: "sales_score"
            }
        },
        'style':  {
            breaks: [0, 250, 500, 750, 1000],
            colors: ["rgb(43,131,186)", "rgb(171,221,164)", "rgb(253,174,97)", "rgb(215,25,28)"],
            zcolors: ["#3aa9e3","#7db5dc","#a8c0d4","#cccccc","#dbb09b","#e2936b","#e3743a"]
        },
        'callback': mycallback
    }
    let myVDV = new VectorDataView(options);

Once the VectorDataView is atached to the target layer, it will retrieve and cook the data everytime it changes, being transparent for the user/dev.

Properties

• map, layer, data, uniqueid, kpi, style,callback, filter: The input options.
• id: unique idenfifier
• dict: the dictionary object of the metadata
• features: array with references to the properties of all the features in the current viewport
• _ready: the dataview has received all the data
• _index: index of the element of the dictionary encoded property used as kpi or -1 if the kpi is a common property.
• _tuplas: object that stores {uniqueid:kpi_value} always in sync with the viewport.

Methods

All the methods relate to the features currently within the viewport, so the map and the dataview are always synced

destroy()

Stop the workers, detach the events and delete al the references, leaving an empty object '{}';

Getters

_getFeatures()

Private, not intended to be used as is. This method is fired internally each time the data changes and:

• rebuilds the features property of the VectorDataview, so there should be no need to use it. It returns the properties of the features in the viewport as an array of objects
• rebuildes the _tuplas array if needed
• applies the filter if any
• re-applies the style if needed
• triggers the callback function

_getDatumIndex(filter)

Private, not intended to be used as is. With a filter or indexas input, it gives back the index of the requested value in a dictionary-encoded property.

Example:

let filter = {
    year: '2015',
    item:'total',
    kpi:'kg_sold'
};
myVDV._getDatumIndex(filter)
// --> 61

_getDatum(array, <filter|index>)

Private, not intended to be used as is. Retrieves the specific value from the array, either using a filter or a numeric index

Example:

let filter = {
        year: '2015',
        item:'total',
        kpi:'kg_sold'
    },
    // random filled array for example sake
    data = [...Array(100)].map(()=>{return Math.random()*37});

myVDV._getDatum(data,filter)
// --> 3.0394481006440444

myVDV._getDatum(data,61)
// --> 3.0394481006440444

_getKPIGrowth(feature, delta)

Private, not intended to be used as is. Retrieves the growth for the KPI related to any other value on any level. Inputs:

• feature: feature object
• delta: object that defines the levelname and steps (default 1 backwards) to compare

Example:

let feature = myVDV.features[0],
    delta = {
        levelName:'year',
        steps: 1
    };

myVDV._getGrowth(feature,delta);
// --> 37

getValues(property <, filter|index, isSource>)

This method retrieves an array the values of the specified property for the features within the viewport. If this property is a dictionary-encoded one, a filter or index argument is needed.

If isSource is set to true, the response will includes all the features which data is available. If this argument is false (default), the response includes only the rendered features (those within BBox)

Example:

myVDV.getValues('merchants')
// --> [8, 21, 5, 9, 20, 17, 3, 8, 15, 30, 9, 10, 7,... ]

let myFilter = {
        'property': 'data'
        'filter': {
            year: '2015',
            item:'apple',
            kpi:'number_of_pies'
            },
        'op': '>=',
        'value': 100
    };

myVDV.getValues('data', myFilter)
// --> [19, 7, 2, 2, 18, 7, 7, 15, 27, 23, 7, 22,... ]

myVDV.getValues('data', 61)
// --> [19, 7, 2, 2, 18, 7, 7, 15, 27, 23, 7, 22,... ]

getAggs(options)

Gives back the most common aggregates for a numeric property within a range or category. If the property is not a number, it will give the count back, but all other aggregations will be set to null.

The input is an object with properties like:

• property: string, name of the property where the filter is goint to be applied to. If it is a dictionary-encoded property, the next property is compulsory:
  • filter: As described above or numeric index
• groupby: The property used for grouping by. Its contents might be numeric and need a numeric range, or string, so it will need a category (string) value. If it is a dictionary-encoded property, the next property is compulsory:
  • groupbyfilter: As described above or numeric index
• range: If groupby is a numeric property, The range can be given as a numeric array [a, b] so the aggregation will be performed with the values X in groupby like a <= X <b. In case it's a string (category), the values of property will be aggregated as per categories.

Example:

let options ={
        'property': 'data'
        'filter': {
            year: '2015',
            item:'apple',
            kpi:'number_of_pies'
            },
        'groupby': 'county',
        'range': 'Orange County'
};
myVDV.getAggs(options)
// --> {count: 3045, min: 110, max: 99802, sum: 82623224, avg: 27134.063711001643}

getChartData(options)

Retrieves aggregated data than can be easyly used in a bar chart. The input is an object with the properties:

• property: The name of the property that stores the Y value to be aggregated per range. If it is a dictionary-encoded property, the next property is compulsory:
  • filter: As described above
• groupby: The name of the property that stores the X value to group features by. If it is a dictionary-encoded property, the next property is compulsory:
  • groupbyfilter: As described above
• ranges: Optional, if null, the method retrieves the available unique categories in the groupby cateogires property
  • Numeric array of breaks like [0, 10, 20, 30, 40, 50]
  • Strings array of categories like ['apples', 'oranges','lemons', 'berries']

It returns an array of objects with the aggregated values per range and the range identificaion.

Example:

let options_numeric ={
    'property': 'data'
    'filter': {
        year: '2015',
        item:'apple',
        kpi:'number_of_pies'
        },
    'groupby': 'data'
    'groupbyfilter':{
        year: '2015',
        item:'apple',
        kpi:'avg_price_kg'
    }
    'ranges': [0, 10, 20, 30, 40, 50]
}
let options_category={
    'property': 'data'
    'filter': {
        year: '2015',
        item:'apple',
        kpi:'number_of_pies'}
    'groupby': 'county'
}

myVDV.getChartData(options_numeric)
// --> [{count: 939, min: 0, max: 996, sum: 17379, avg: 18.507987220447284, range: [10, 20]},...]

myVDV.getChartData(options_category)
// --> [{count: 866, min: 0, max: 60675119, sum: 1928986525, avg: 2227467.118937644,category:"Orange County"},...]

getRanking(items)

Returns an array of sorted objects that can be used to build a ranking in the UI. It needs to be feeded with an array of objects that defines the items in the output.

• Option A: property value

  • name: the name of the value to be given
  • property: The name of the property that stores the value. If it is a dictionary-encoded property, the next property is compulsory:
  • filter: As described above
  • isDefaultSort: boolean to set one of the properties as sorting criterium (desc)

• Option B: KPI growth:

  • name: the name of the value to be given
  • delta (as described in _getGrowth)
  • isDefaultSort: boolean to set one of the properties as sorting criterium (desc)

NOTE: If the desired property is the currently active KPI, just use

• name: the name of the KPI to be given
• property: kpi_value

Example:

let items = [
    {
        'name': 'Store',
        'property': 'storename'
    },
    {
        'name': 'Main score',
        'property': 'kpi_value',
        'isDefaultSort': true
    },
    {
        'name': 'Total sold',
        'property': 'data',
        'filter': {
            year: '2018',
            item:'total',
            kpi:'kg_sold'
        }
    },
    {
        'name': 'Yearly growth',
        'delta': {
            'levelName': year,
            'steps': 1
        } 
    }
];
myVDV.getRanking(items)
// --> [{'id': 1231246512,  'Store': 'Natural Food', 'Total sold': 56330, 'Yearly growth: -37'}, ...]

getFeature(uniqueidvalue)

Retrieves the full feature.properties object for a given unique id.

Example:

myVDV.getFeature(360610171006000);
// --> {id: 360610171006000, area: 20944, merchants: 8, data: Array(700),...}

getSeries(options)

Gets a series of values along any level in the dictionary (X-axis). So, if the X-axis describes date values it will return a time series for the KPI, but if X-axis is a category branch, this method retrieves the values of all the KPIs per category.

The results can be requested per unique feature or aggregated values for all the features in the bounding box. The response is an array of objects {x, y}, being y a plain value when an unique feature is requested or an object with aggregated values {min, max, sum, avg}.

It keeps the values set by SetKPI for all the other branches

Input, an object with the next properties:

• x_axis: level name of the dictionary to be used as X-axis in the series
• y_axis: level name of the dictionary too be used as Y-axis in the series
• ỳ_axis_filter: filter function for Y values, defaults to a => true, so it gets all the available Y values
• id: Optional, if we need the series for an uniique feature

Example:

myVDV.getSeries( {'x_axis': '0_dates', 'y_axis': '2_scores'})
// --> [{x":"2012-02-01","y":{kpi_1:{"min":0,"max":999,"sum":261106,"avg":701.8978494623656}, kpi_2:{...}},...]

myVDV.getSeries({'id':360470519003007, 'x_axis': '0_dates', 'y_axis': '2_scores', 'y_axis_filter': (a=> a === 'total_sales')})
// --> [{"x":"2012-02-01","y":0},...]

Setters

setKPI(kpi)

Sets the KPI the app will be focused on, and forces the refresh of the data and styling. The kpi object is defined by the name of the property and a filter if needed. Returns the VectorDataView object.

Example:

let mykpi = {
    'property': 'data',
    'filter': {
        year: '2018',
        item:'total',
        kpi:'kg_sold'
    }
};
myVDV.setKPI(mykpi);
//--> myVDV

setFilter(filters)

This method filters the layer in the map, and the vectordataview is therefore filtered too. Internally, it makes use of Mapbox specification for expresions. Its input is an array of filters that will be all required to be fulfilled. Each filter is an object with:

• property: string, name of the property where the filter is goint to be applied to. If it is a dictionary-encoded property, the next property is compulsory:
  • filter: As described above
• op: string with the comparision operator
• value: value to be compared with

Returns the VectorDataView object.

Example:

let myFilters = [{
        'property': 'data'
        'filter': {
            year: '2015',
            item:'apple',
            kpi:'number_of_pies'
            },
        'op': '>=',
        'value': 100
    },{
        'property': 'the_answer_to_life_the_universe_and_everything'
        'op': '=='
        'value': 42
    }];
myVDV.setFilter(myFilters)

setStyle(style)

Set the style of the map view based on the KPI and a style expresions related to a variable called _kpi to be applied to an styling property. Returns the VectorDataView object

Example:

setStyle({
    property: 'fill-opacity',
    expr: `["*", 0.25, ["var","_kpi"]]`
})

setZStyle(boolean)

Switch the style to a dynamic viewport-related z-score based choropleth

Utils

window.WorkerFromFunction(func, autoclose)

Returns a webworker object from a reference to a function. Input:

• func: reference to a function
• autoclose: boolean to force the worker to close itself once the function has run

Once defined, the parameters to the function shuld be passed as an array of arguments in the postMessage

Example:

let discount = (new_price, old_price) => (round(100 * (old_price - new_price) / old_price))+'%';
let myworker = WorkerFromFunction(discount);
myworker.onmessage(e =>{console.log(e.data)})
myworker.postMessage([85, 120]);
// console: '29%'

window.makeUUID()

Generates an UUID compliant with RFC4122 v.4

Example:

makeUUID();
// --> "5e5257b3-d81a-4c26-ac44-427afdbc195d"