koop-provider-google-analytics

Getting Started
Service endpoint pattern
Sample requests

Getting Started

Setup

This provider ships with a sample instance server.js which registers koop-provider-google-analytics. For the provider to work properly a few variables must be set for the environment or in a configuration file.

Environment Variables

|Variable|Description|Required| |---|---|---| | GOOGLE_VIEW_ID | The unique ID used to retrieve the Google Analytics data. Available through your Google Analytics dashboard. | Yes | | GOOGLE_ANALYTICS_TIMEZONE | String representing the timezone used to define the Google Analytics view/report data. See moment/timezone for valid timezones. | Yes | | GOOGLE_CLIENT_EMAIL | The Google Analytics client email generated by Google for connecting to the API. | Yes | | GOOGLE_PRIVATE_KEY | The Google Analytics private key generated by Google for connecting their API. Must be base64 encoded (due to multiline key)| Yes | | GOOGLE_START_DATE | Beginning of the time range used for analytics reports in YYYY-MM-DD format. Defaults to 2005-01-01. | No | | ANALYTICS_CACHE_TTL | Number of seconds to cache analytics data until re-retrieved from Google Analytics. If not set, there will be no caching. | No |

Config file

If you prefer, you can set the required variables in the Koop configuration file rather than as environment variables. See config/default.js.example for specifics. The configuration file can also be use to customize the provider's support for additional Google Analytics dimensions and metrics. The config/default.js.example includes examples of adding dimensions and metrics not already defined in src/constants-and-lookups.js. You will need to remove .example from the filename in order for the config npm to read and register this file.

Backfill multi-dimesional time-series data

The Google Analytics API won't include empty rows when requesting multiple dimensions; if you want empty rows for a request with a time dimension plus at least one other dimension, set the backfillTimeseries configuration setting to true.

With the above environment variables and or config set, the Koop server can be started with:

  node server.js

The Koop API will be listening on port 8080.

Service endpoint pattern

The service endpoint conforms to the following pattern:

http://<domain>/google-analytics/:id/FeatureServer/0/query?<query-parameters>

:id parameter

The :id parameter is composite of three other parameters. The requested metric, dimensions, and transformation options are delimited like: <metric>:<dimensions>~<options>. A single metric is required. Multiple dimensions or options can be delimited with ,. You can skip using dimensions or options by leaving out the delimiters and values. For example, to request a metric without dimensions but still add options, the :id parameter would look like <metric>~<options>. See each parameter below for details.

metric

The metric parameter indicates the requested metric. Only a single metric can be requested. By default, the provider is configured to allow metrics in the following table. Additional metrics can be added with the configuration file.

|value|description| |---|---| |views|Number of page views| |uniqueViews|Number of unique page views| |sessions|Number of page views| |totalEvents|Number of total events|

dimensions

The dimensions parameter indicates the requested dimension(s) for slicing the data. Multiple dimensions can be requested by concatenating values with ,, e.g month,country. Default dimension types are listed below. Additional dimensions can be added with the configuration file.

|value|description| |---|---| |hour| Slice data by timestamp in hourly intervals | |day| Slice data by timestamp in daily intervals | |week| Slice data by timestamp in weekly intervals | |month| Slice data by timestamp in monthly intervals | |country| Slice data by country | |eventCategory| Slice data by eventCategory | |eventAction| Slice data by eventAction | |eventLabel| Slice data by eventAction | |hostname| Slice data by hostname |

Query parameters

Query parameters further refine the metrics request and are optional. Below are a list of the currently supported query parameters and their default values:

|name|type|description|default| |---|---|---|---| |time|string|Comma separated date/date-time range for the requested metrics. Can be unix timestamp or YYYY-MM-DD strings, e.g. 2017-01-01,2018-01-01 or 1483257600,1514793600. Use "null" to omit part of the range.| 2000-01-01 to current date | |where|string|A SQL style WHERE clause. See notes below.||

where parameter rules

This provider converts the SQL found in the where parameter to arrays of Google Analytics metric and dimension filter clauses. Unfortuntately, some wheres will not be translatable to filter clauses due to Google Analytics business rules. The provider will give informative errors when a where cannot be translated, but general guidelines are provided below:

1. Predicates of a given type (metrics or dimensions) cannot be combined with more than one type of logical operator. For example, view > 100 OR uniqueViews > 10 AND sessions > 5 will not work.
2. Logical combination of metric and dimension predicates must be with AND, e.g. (sessions > 100) AND (country = 'Canada').
3. Currently supported operators for metric predicates include =, <, >. Combine predicates with OR to achieve <= or >=, e.g. (sessions > 100 OR sessions >= 100).
4. The only supported operator for dimension predicates is =.
5. Complex wheres that include multiple metric and dimension predicates should be partitioned by type (metric/dimension) with parenthesis, e.g (sessions > 100 OR sessions >= 100) AND (country = 'Canada' OR country = 'Mexico').

Feature Service query parameters

Since Koop employs the FeatureServer output service, you can use its subset of the ArcGIS REST API parameters for feature service layers.

Sample requests

1. Monthly timeseries of page views for date range
2. Sum of all page views for date range
3. Average monthly page views for date range
4. Sum of total events dimensioned by event category for a date range
5. Top ten session counts by country for a date range
6. Multiple metrics, multiple dimensions
7. Dimension by eventCategory and filter with where

*NOTE: All request require a token parameter or authorization header with a valid user token.

Monthly timeseries of page views for date range

http://localhost:8080/google-analytics/views:month/FeatureServer/0/query?time=2017-01-01,2018-07-01

Response:

{
    "objectIdFieldName": "OBJECTID",
    "globalIdFieldName": "",
    "hasZ": false,
    "hasM": false,
    "spatialReference": {
        "wkid": 4326
    },
    "fields": [
        {
            "name": "OBJECTID",
            "type": "esriFieldTypeOID",
            "alias": "OBJECTID",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        },
        {
            "name": "timestamp",
            "type": "esriFieldTypeDate",
            "alias": "timestamp",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null,
            "length": 36
        },
        {
            "name": "views",
            "type": "esriFieldTypeInteger",
            "alias": "views",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        }
    ],
    "features": [
        {
            "attributes": {
                "timestamp": 1485907199999,
                "views": 898118,
                "OBJECTID": 1003353633
            }
        },
        ...
        ...
        ...
        {
            "attributes": {
                "timestamp": 1533081599999,
                "views": 26983,
                "OBJECTID": 1924415847
            }
        }
    ],
    "exceededTransferLimit": false
}

Sum of all page views for date range

http://localhost:8080/google-analytics/views/FeatureServer/0/query?time=2017-01-01,2018-07-01

Response:

{
    "objectIdFieldName": "OBJECTID",
    "globalIdFieldName": "",
    "hasZ": false,
    "hasM": false,
    "spatialReference": {
        "wkid": 4326
    },
    "fields": [
        {
            "name": "OBJECTID",
            "type": "esriFieldTypeOID",
            "alias": "OBJECTID",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        },
        {
            "name": "views",
            "type": "esriFieldTypeInteger",
            "alias": "views",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        }
    ],
    "features": [
        {
            "attributes": {
                "views": 28411690,
                "OBJECTID": 952929127
            }
        }
    ],
    "exceededTransferLimit": false
}

Average monthly page views for date range

http://localhost:8080/google-analytics/views:month/FeatureServer/0/query?time=2017-01-01,2018-07-01&outStatistics=[{"statisticType": "avg","onStatisticField": "views","outStatisticFieldName": "average_monthly_views"}]

Response:

{
    "displayFieldName": "OBJECTID",
    "fields": [
        {
            "name": "average_monthly_views",
            "type": "esriFieldTypeDouble",
            "alias": "average_monthly_views",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        }
    ],
    "features": [
        {
            "attributes": {
                "average_monthly_views": 1687865.6923076923
            }
        }
    ]
}

Sum of total events dimensioned by event category for a date range

http://localhost:8080/google-analytics/totalEvents/eventCategory/FeatureServer/0/query?time=2017-01-01,2018-07-01

Response:

{
    "objectIdFieldName": "OBJECTID",
    "globalIdFieldName": "",
    "hasZ": false,
    "hasM": false,
    "spatialReference": {
        "wkid": 4326
    },
    "fields": [
        {
            "name": "OBJECTID",
            "type": "esriFieldTypeOID",
            "alias": "OBJECTID",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        },
        {
            "name": "eventCategory",
            "type": "esriFieldTypeString",
            "alias": "eventCategory",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null,
            "length": 128
        },
        {
            "name": "totalEvents",
            "type": "esriFieldTypeInteger",
            "alias": "totalEvents",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        }
    ],
    "features": [
        {
            "attributes": {
                "eventCategory": "API",
                "totalEvents": 9503,
                "OBJECTID": 2141521907
            }
        },
        {
            "attributes": {
                "eventCategory": "API Explorer",
                "totalEvents": 35130,
                "OBJECTID": 339333549
            }
        },
...
...
...
        {
            "attributes": {
                "eventCategory": "widgets",
                "totalEvents": 160570,
                "OBJECTID": 1460530564
            }
        }
    ],
    "exceededTransferLimit": false
}

Top ten session counts by country for a date range

http://localhost:8080/google-analytics/sessions/country/FeatureServer/0/query?time=2017-01-01,2018-07-01&orderByFields=sessions%20DESC&limit=10

Response:

{
    "objectIdFieldName": "OBJECTID",
    "globalIdFieldName": "",
    "hasZ": false,
    "hasM": false,
    "spatialReference": {
        "wkid": 4326
    },
    "fields": [
        {
            "name": "OBJECTID",
            "type": "esriFieldTypeOID",
            "alias": "OBJECTID",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        },
        {
            "name": "country",
            "type": "esriFieldTypeString",
            "alias": "country",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null,
            "length": 128
        },
        {
            "name": "sessions",
            "type": "esriFieldTypeInteger",
            "alias": "sessions",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        }
    ],
    "features": [
        {
            "attributes": {
                "country": "United States",
                "sessions": 3465985,
                "OBJECTID": 2115310799
            }
        },
        {
            "attributes": {
                "country": "Canada",
                "sessions": 344534,
                "OBJECTID": 1164473059
            }
        },
        {
            "attributes": {
                "country": "United Kingdom",
                "sessions": 259762,
                "OBJECTID": 1199712002
            }
        },
        {
            "attributes": {
                "country": "Colombia",
                "sessions": 164969,
                "OBJECTID": 463997413
            }
        },
        {
            "attributes": {
                "country": "Japan",
                "sessions": 111078,
                "OBJECTID": 660706644
            }
        },
        {
            "attributes": {
                "country": "Brazil",
                "sessions": 104631,
                "OBJECTID": 1916301181
            }
        },
        {
            "attributes": {
                "country": "Australia",
                "sessions": 100033,
                "OBJECTID": 200827751
            }
        },
        {
            "attributes": {
                "country": "New Zealand",
                "sessions": 70347,
                "OBJECTID": 266670080
            }
        },
        {
            "attributes": {
                "country": "Spain",
                "sessions": 52992,
                "OBJECTID": 1375456220
            }
        },
        {
            "attributes": {
                "country": "France",
                "sessions": 50162,
                "OBJECTID": 1909896837
            }
        }
    ],
    "exceededTransferLimit": false
}

Multiple metrics, multiple dimensions

http://localhost:8080/google-analytics/sessions::views/country::eventCategory/FeatureServer/0/query?time=2017-01-01,2018-07-01

{
    "objectIdFieldName": "OBJECTID",
    "globalIdFieldName": "",
    "hasZ": false,
    "hasM": false,
    "spatialReference": {
        "wkid": 4326
    },
    "fields": [
        {
            "name": "OBJECTID",
            "type": "esriFieldTypeOID",
            "alias": "OBJECTID",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        },
        {
            "name": "country",
            "type": "esriFieldTypeString",
            "alias": "country",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null,
            "length": 128
        },
        {
            "name": "eventCategory",
            "type": "esriFieldTypeString",
            "alias": "eventCategory",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null,
            "length": 128
        },
        {
            "name": "sessions",
            "type": "esriFieldTypeInteger",
            "alias": "sessions",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        },
        {
            "name": "views",
            "type": "esriFieldTypeInteger",
            "alias": "views",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        }
    ],
    "features": [
        {
            "attributes": {
                "country": "Australia",
                "eventCategory": "dataset",
                "sessions": 3,
                "views": 7,
                "OBJECTID": 1105561907
            }
        },
        ...
        ...
        ...
        {
            "attributes": {
                "country": "United States",
                "eventCategory": "Search",
                "sessions": 716,
                "views": 10475,
                "OBJECTID": 2012216286
            }
        }
    ],
    "exceededTransferLimit": false
}

Dimension by eventCategory and filter with where

http://localhost:8080/google-analytics/sessions::views/eventCategory/FeatureServer/0/query?where=(country='United States') AND views > 999

{
    "objectIdFieldName": "OBJECTID",
    "globalIdFieldName": "",
    "hasZ": false,
    "hasM": false,
    "spatialReference": {
        "wkid": 4326
    },
    "fields": [
        {
            "name": "views",
            "type": "esriFieldTypeInteger",
            "alias": "views",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        },
        {
            "name": "eventCategory",
            "type": "esriFieldTypeString",
            "alias": "eventCategory",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null,
            "length": 128
        },
        {
            "name": "sessions",
            "type": "esriFieldTypeInteger",
            "alias": "sessions",
            "sqlType": "sqlTypeOther",
            "domain": null,
            "defaultValue": null
        }
    ],
    "features": [
        {
            "attributes": {
                "sessions": 80831,
                "views": 157629,
                "eventCategory": "ArcGIS"
            }
        },
        ...
        ...
        {
            "attributes": {
                "sessions": 813,
                "views": 18397,
                "eventCategory": "Search"
            }
        }
    ],
    "exceededTransferLimit": false
}