Metric Server

Metric.im server for pinging and reporting event data.

Usage

npm install @metric-im/metric-server

The module can be integrated into host and invoked directly

import MetricServer from '@metric-im/metric-server';
// Get the application interface
const metricApi = await MetricServer.getApi(mongo.db(),{}); // insert relevant database reference
// Record an event with ping([body])
let eventBody = metricApi.initializeEvent('myAccount',req); // Optionally parse web request. Account name is arbitrary
await metricApi.ping(Object.assign(eventBody,{hi:"hello",color:"green"})); // Ping body is context and custom data
// Retrieve event data with pull([account],[path],[options])
let account={id:'myAccount',super:true}; // create a stub account to bypass metric access control
let results = await metricApi.pull(account,req.params[0],req.query,res); // path identifies format, namespace, dimensions & metrics

The body sent to ping can be anything, though complex objects are less useful than flat objects. All events are recorded with a timestamp, namespace, account and unique identifier. Additional attributes are arbitrary. The namespace helps interpret the attributes and generate derived data, but is not required.

The path identifies the output format (such as raw json or a pie chart), namespace, dimensions and metrics. Dimensions organize the results into "columns" while metrics perform aggregate math on the data set that satisfies all dimensions, a "row".

Please refer to the Reference Guide for details, syntax and options.

Deployment Notes

This package relies on the metric-im/wiki-mixin which requires JRE for graphics. This dependency is not important to metric-server and will be removed.

sudo apt-get install -y openjdk-8-jdk
sudo apt-get install -y graphviz

The package.json references dotenv. Install dotenv globally

sudo npm i -g dotenv

The event collection should have a wildcard index. We don't know which attributes are going to be used as dimensions, making them keys. The event id (_id), time stamp (_time), and namespace (_ns) are commonly used.

metric> db.event.getIndexes()
[
  { v: 2, key: { _id: 1 }, name: '_id_' },
  { v: 2, key: { '$**': 1 }, name: '$**_1' },
  { v: 2, key: { _ns: 1 }, name: '_ns_1' },
  { v: 2, key: { _time: 1 }, name: '_time_1' }
]

Reference Guide

The metric server is essentially an abstraction of Mongo's rich aggregation waterfall applied to an arbitrary set of atomic events. An event has an ID and a Time Stamp. All other data is whatever needs to be recorded to capture the relevant context of the event. In addition to event recording (ping) and event analysis and reporting (pull), there are a few additional methods to maintain the collection.

• ping() - Record an event including an arbitrary set of data points
• pull() - Query the event collection to produce raw data, charts or tables
• initializeEvent() - An event triggered with a web call is parsed for context such as user agent and IP (which is not recorded)
• ontology() - Manage the ontology that defines the namespaces and field handling
• redact() - Remove an event. Avoid changing the event history whenever possible
• analysis() - Sample the event collection to examine the fields available, some values and the interpreted data types

NOTE: Ontology, redact and analysis are not yet available through the imported Api, only the web routes

Ping()

syntax: PUT|GET /ping/{format}/{namespace}[?[attribute=value]]

| name | description | |---|---| | format | Determines the return type of the request. See Ping Return Formats | | namespace | Identifies the event namespace. This governs access and attribute data types | | attribute=value | The query string can include any number of attribute value pairs to be queried with pull |

Using PUT, attributes can be provided through the body rather than query string. This is preferable because integers and floats can be passed, rather than just strings. Declared attributes will be cast into the specified data type regardless.

The body of a PUT may be provided as an array, in which case each array entry is treated as an individual event.

Ping Return Formats

• silent - returns 204 success status with no payload
• pixel - returns a 1x1 transparent png
• script - returns an empty payload with application/javascript mime type
• json - returns the object recorded for the event
• table - returns an html table of name value pairs

Any other tag will return an empty 204, same as silent.

System Values

_time, _ns and _account are added automatically. If _time is provided as an event attribute, it will override the default.

If the namespace selects any refiners, each is executed with the base event object and the request context. This context includes hostname, url, ip and ua (user agent). These values can be provided explicitly by defining _origin as attribute. It is expected to be an object with one or more of these context values defined. _context, if present is removed from the event record before being written.

Ping Examples

https://metric.im/ping/silent/navigation?site=hello.com&path=/about

Pull()

syntax: GET /pull/{format}/{namespace}/{dimension[:{qualifier}]}/[{metrics[:aggregation]}][?{option}[&{option}]]

| name | description | |---|---| | format | Determines the format of the result. This includes: json, table, csv and chart | | namespace | Identifies the event namespace. This governs access and attribute data types | | dimensions | Comma separated list of one or more dimensions. Dimension values organize results for metric aggregation | | qualifier | A dimension can be qualified by following it with a colon and a limiter. The simplest is an explicit match value, but ranges and even mongo statements can be used. | | metrics | Metrics are the attribute values that are measured. Metrics should be declared in to specify the aggregation method and display formatting. | | option | Options to modify the results |

Aggregation methods

The metrics for each record that matches the given dimensions is run the a formula to produce a single result. The built in formulas are sum, avg, first, last, min, max, etc. These are define by mongo. See MongoDB group accumulators

In addition, Metric.im supports custom accumulators that can be defined for a namespace.

Options

| option | default | description |
| --- | --- |---| | sort | _id | sort can be a field name or a mongo sort object, such as {modified:-1}. Note that all _id fields are natively date sorted. | | days | | limit the results to look back only the specified number of days. | | since | | identify only records since a give date specified as YYYY-MM-DD. "Since" can be used in conjunction with "days" | | where | | apply a match statement to the request. The value should be in mongo syntax. This can usually be done by qualifying dimensions. where={name:"Alice"} is equivalent to /name:Alice in the dimension path. |

Formats

| name | mimetype | description | | --- | --- |---| | json | application/json | raw object format ideal for programmatic results | | csv | text/plain | comma separated grid for use with spreadsheets | | table | text/html | HTML page with embedded TABLE element. Includes simple styling | | chart | text/html | HTML page with embedded chartjs object. To select a chart type other than bar, use dot notation. i.e chart.pie. | | map | text/html | HTML page with a google map object. The dataset must include latitude and longitude. The rendering is a marker for each object, but with dot notation heatmap or polyline path can be rendered |

See https://chartjs.org for more information about chart rendering. Metric flips and adjusts data to adjust to match the chart type. The following types render relevant results:

• chart[.bar] (default)
• chart.line
• chart.pie
• chart.doughnut
• chart.radar
• chart.polarArea

For map formats, the result set must include attributes for latitude and longitude (dimensions or metrics). Any other attributes are inserted as hover text. The following rendering options are available

• map[.marker] - puts a pin in the map
• map.path - trace a line from each point
• map.heat - visualize the concentration of points

Common Format options.

• .nolegend - Hides the label text on graphs. For example, /chart.bar.nolegend
• .file - returns a file with the results rather than displaying on screen. Not available for graphs or maps.

Pull examples

Number of posts by day

/pull/chart.bar.nolegend/sezus/date/_count?sort=date&where={action:create}

Reactions

/pull/chart.line/sezus/date/like,dislike,important,badFaith,troll,insightful,misleading

Activity by country

/pull/chart.pie/sezus/country/_count?sort=_count:-1&limit=8

Geographic Activity (activity centered in a country indicates we have no further detail)

/pull/map.heat/sezus/_id/latitude:last,longitude:last

Total number of posts... all events where action is 'create'

/pull/json/sezus/action:create/_count
--or--
/pull/json/sezus/action/_count?where=action:create

initializeEvent()

InitializeEvent is a convenience function. It reads the expressjs request object for context data. Source data, such as the user agent string and IP address, is placed in the _origin block of the event. This can be used by the refiners, but is not stored with the event record when PII is a concern.

ontology()

The ontology defines the meta data behind the namespaces and attribute. This is generally used only by the system.

syntax: GET /ontology/ns[/{namespaceId}][/fields]

Without specifying namespaceId, the request returns all namespaces explicitly available to this account (excluding public namespaces). By providing a namespaceId the result is the full namespace record and the fields explicitly defined to that namespace.

Append the query with /fields and the result is all the attributes, including inherited attributes that are defined to this namespace.

redact()

syntax: GET /redact/{namespace}[/id,...][?{since}]

Remove events from the system

| name | description | |---|---| | namespace | Identifies the event namespace. This governs access and attribute data types. You must have admin privileges to a namespace to remove events | | id | Optionally provide one or more explicit event id's, separated by commas. This will remove the specifically identified events | | since | enter any combination of minutes=value, hours=value and or days=value to instruct redaction of events newer than now minus the time specified |

analysis()

syntax: GET /analysis/{namespace}

Analysis is a tool to assist in writing pull requests. It is a schema of the data derived from a sample of events. Some event fields will be named and typed in the Ontology. If not, the engine guesses the datatype based on the existing values.

Design Concepts

Refinery and Plugins

Refiners Available

Ontology Editing

Field Specifications

Derived fields

Namespace Inheritance

Named Queries

Optimization

Event Refinery

Event Refinery

Reserved Attribute Names

• year - YYYY
• monthName - January, February, ...
• month - 1-12
• day - 1-31
• hour - 0-60
• minute - 0-60
• second - 0-60
• week - 1-52
• weekday - Sunday, Monday, ...
• date - YYYY-MM-DD
• _time - now
• _ns - event name space or "ping"
• _account - authorized account or "public"