The mars-sdk

A library of utilities intended to make the development of collector node.js script both easier, and more robust.

The core utilities can be required and used in the simple form:

const utils = require('@moogsoft/mars-sdk');

utils.debug('Mooo');

Or in the prefered deconstructed method:

const {
    debug,
} = require('@moogsoft/mars-sdk');

debug('Mooo');

Extended utilities can be loaded as follows:

const rest = require('@moogsoft/mars-sdk/rest');
const Event = require('@moogsoft/mars-sdk/Event');

Utility Reference

Core Utilities

Loggers

• debug
• info
• warn
• error

Discovery Utilities

• fileExists
• canRun
• hasCmd
• procRunning
• DiscoveryResult
• Reason
• constants
• sendDiscovery

Configuration Utilities

• getConfig
• getCredentials
• exportConfig
• getMarDir

Shedulers

• registerScheduled
• carousel

API Object builders and formatters

• Metric
• Event
• Bitmask
• sendResult
• sendMetrics
• sendEvents

General Purpose Utilities

• exec
• JSONToKv
• dehumanize
• jsonParse
• isInFilters
• passFilter

Extended Utilities

REST

• REST

debug

debug(string)

Writes a JSON formatted string to stdout, with a "level" key set to "debug" and the supplied string as the value for "msg", which is processed by the collector as a log message.

info

An "info" level log message. See debug for details

warn

A "warn" level log message. See debug for details

error

An "error" level log message. See debug for details

jsonParse

jsonParse(string)

A wrapper for JSON.parse()

• Returns: <Object> Parsed object

fileExists

fileExists(filePath)

Checks to see if a given file exists

• filePath <String>
• Returns: <boolean>

canRun

canRun(cmd)

Checks to see if a given command is executable

• cmd <String>
• Returns: <boolean>

hasCmd

hasCmd(cmd)

Checks to see if a given command exists, and is executable by the current user

• cmd <String>
• Returns: <boolean>

procRunning

procRunning(proc)

Checks to see if a given process/task (name) is running

• proc <String> Name of processes
• Returns: <boolean>

DiscoveryResult

const DiscoveryResult = require('@moogsoft/mars-sdk/DiscoveryResult');
const result = new DiscoveryResult()
    .<setMethod(value)>
    [...]

An external class to generate a well formed discovery result object.

• Returns: <Object>
• Methods:
  • .setMoob(moob) Required (If single)
    • moob: <String> The Managed Object that has been enabled
  • .setMoobs(moobs)Required (If multiple)
    • moob: <Object> A list of Managed Objects that have been enabled (if multiple are suported)
  • .setReason(reason) Optional
    • reason: <Object> A well formed reason object built with the Reason method
  • .setActive(active)
    • active <boolean>

Examples:

Initialize and turn on a MAR:

let builder = new DiscoveryResult.setMoob('system').setActive(true);

Disable a MAR based on subsequent logic:

builder = builder.setActive(false);

Reason

An external class to generate a well formed Reason to be used by the DiscoveryResult.setReason() method

• Returns: <Object>
• Methods:
  • .setRecoverable(recoverable)
    • recoverable: <boolean> Is it completely undiscoverable, or should the user be prompted for miising information?
  • .setMsg(msg)
    • msg: <String> A message that will be presented to the user
  • .setType(type)
    • type: <String> A categorization of the type of discovery "failure". While this can be any string, the following categores must be set from the constants module:
      • INSUFFICIENT_PRIVILEGES
      • INVALID_CREDENTIALS
      • MISSING_CREDENTIALS
      • MISSING_PROCESS
      • MISSING_COMMAND

Examples:

Disable a MAR because a process was not detected:

const {
    MISSING_PROCESS,
    MISSING_COMMAND,
}  = require('@moogsoft/mars-sdk/constants');

builder = builder.setActive(false)
    .setReason(new Reason()
        .setRecoverable(false)
        .setMsg('redis server process not found')
        .setType(MISSING_PROCESS);

constants

The constants module exposes common constants used during discovery and collection. Available constants:

INSUFFICIENT_PRIVILEGES
INVALID_CREDENTIALS
MISSING_CREDENTIALS
MISSING_COMMAND
MISSING_PROCESS
COUNTER
GAUGE

sendDiscovery

sendDiscovery(discovery)

Writes a JSON string to stdout containing the results of the disovery, which is interpretted by the collector

• discovery <Object> A well formed "disovery" object

getConfig

getConfig()

• Returns: <Object> An object containing configuration (if any) for that MAR, for that collector instance

getCredentials

getCredentials()

• Returns: <Object> An object containing credentials (if any) for that MAR, for that collector instance

exportConfig

exportConfig(config)

Writes a JSON string to stdout containing the config (see getConfig(), which is interpretted by the collector

• config <Object>

getMarDir

getMarDir()

Returns the absolute path of the MAR script that is being execute. Useful for accessing external files/artifacts in the MAR

• Returns: <String>

registerScheduled'

registerScheduled(func)

Takes a function and calls if if the first input argument is the functions name. Useful when calling separate functions on different schedules for a scheduled collector.

• func: <function> A collector function that generates events/metrics

carousel'

carousel(interval, func, args)

Takes an interval, function of arity 1, and a list of arguments to pass to a function. Calls the function with each argument in turn, spreading the calls across the supplied interval. (A form of load balancer). e.g: If the interval is 10s, and there are 10 args, the function will be executed every second.

• interval: <Number> Time interval in ms
• func: <Function> The function to be executed
• args: <Array> The list of arguments

Metric

An "extended" class to generate a well formed Metric to be used by the sendMetrics() method

• Returns: <Object>
• Methods:
  • .setData(data)
    • data: <Number> a time series value to be observed
  • .setMetric(metric)
    • metric: <String> The name of the metric or "check". (e.g: NetworkIO)
  • .setSource(source)
    • source: <String> the unique name, or "namespace" of the resource the metric is associated with
  • .setKey(key)
    • key: <String> A sub key of the resource (e.g: the network interface)
  • .setTime(ts)
    • ts: <unixtime> A unix timestamp in ms. The collector will provide on if not set
  • .setDescription(desc)
    • desc: <String> A description of the metric
  • .setUtcOffset(offset)
    • offset: <String> utc_offset The timezone of the metric, defaults to the current timezone
  • .setAdditionalData(ad)
    • ad: <Object> Any addition JSON metadata that is not suited for tags, but might be useful for upstream processing or for a user
  • .setTags(tags)
    • tags: <Object> An object, containing a list/array of key:value pairs
  • .setType(ad)
    • type: <String> The metric type, one of either counter or gauge, defaulting to gauge
  • .setUnit(unit)
    • unit: <String> The metric unit. e.g: %, + (non-negative), or a unit like kb. Used by the UI for chart treatment
  • .setWindow(window)
    • window: <Number> The window value
  • .gauge()
    • Sets the metric type to gauge
  • .counter()
    • Sets the metric type to counter
  • .from(object)
    • object: <Object> Create a Metric from an object. The source object is expected to have the same property names as the Metric. Missing properties will be ignored (left as the default undefined).
      • data
      • metric
      • source
      • key
      • utc_offset
      • description
      • additional_data
      • tags
      • type
      • unit
      • window
      • time

Example

const Metric = require('@moogsoft/mars-sdk/Metric');

const metric = new Metric()
    .setMetric('duration')
    .setSource(`${jname}-${stage.name}`)
    .setData(stage.durationMillis)
    .setTime(stage.startTimeMillis)
    .setTags({ job: job.id })
    .setAdditionalData({
        status: stage.status,
        href: encodeURI(stage._links.self.href),
    });

Event

An "extended" class to generate a well formed Event to be used by the sendEvents() method

• Returns: <Object>
• Methods:

  • .setSeverity(severity)

    • severity: <String> OR <Number> Can be one of the following

    | enum | String | Notes | | ---- | ------ | ----- | | 0 | CLEAR | Clears a (mandatory) preceding event severity | | 1 | UNKNOWN | The severity cannot be determined, usually indicates a problem, or unrecognized event | | 2 | WARNING | | | 3 | MINOR | | | 4 | MAJOR | | | 5 | CRITICAL | |

  • .setSource(source)

    • source: <String> the unique name, or "namespace" of the resource the metric is associated with

  • .setCheck(check)

    • check <String> The check that failed to generate this event, for example "cpu load"

  • .setDescription(description)

    • description <String> A description of the event

  • .setTime(time)

    • time: <unixtime> The timestamp of the event in s, if omitted defaults to the current time

  • .setUtcOffset(utc_offset)

    • utc_offset: <String> The UTC offset of the data, for example +01:00, defaults to the current TZ

  • .setDedupeKey(dedupe_key)

    • dedupe_key: <String> [Optional] The deduplication key of this event, used when processing into an Incident

  • .setManager(manager)

    • manager: <String> [Optional] The manager of this event

  • .setService(service)

    • service: <String> [Optional] An array of services impacted by this event

  • .setAlias(alias)

    • alias: <String> [Optional] An alias for this event

  • .setClass(clazz)

    • clazz: <String> [Optional] The class of this event, for example Storage, AWS, Network, etc

  • .setTags(tags)

    • tags: <Object> [Optional] Key-value pairs of metadata for the event

  • .from(object)

    • object: <Object> Create an Event from an object. The source object is expected to have the same property names as the Event. Missing properties will be ignored (left as the default undefined).
      • severity
      • source
      • check
      • description
      • time
      • utc_offset
      • dedupe_key
      • manager
      • service
      • alias
      • class
      • tags

Bitmask

An "extended" class to generate a well formed Bitmask metric to be used by the sendMetrics() method

• Returns: <Object>
• Methods:
  • .setKeys(keys)
    • keys: <Array> An array of strings, represnting the keys, or lables of the binary representation
  • .setValues(values)
    • values: <Array> An array of boolean values (corresponding to the keys)
  • .addValue(key, value) Appends a key-value pair to the value array

sendResult

sendResult(result)

Writes a JSON string to stdout returning a SINGLE datum, which is interpretted by the collector

• result <Object> A well formed "Metric" object

sendMetrics

sendMetrics(metrics)

Writes a JSON string to stdout returning a batch of metrics, which is interpretted by the collector

• metrics <Array> An array of well formed "Metric" objects

sendEvents

sendEvents(events)

Writes a JSON string to stdout returning a batch of events, which is interpretted by the collector

• events <Array> An array of well formed "Event" objects

exec

exec(cmd, args)

Executes a shell process

• cmd: <String> A command in the the collector execution path, or an absolute path
• args: <Array> An array of arguments to the cmd
• Returns: <Object>:
  • status: <Number> shell return value
  • stdout: <String> Text that was written to stdout
  • stderr: <String> Text that was written to stderr

JSONToKv

JSONToKv(object, array, func)

Builds an array of key-value pairs from nested JSON/BSON, by "flattening" the nested structure, concatenating the keys with an underscore

• object: <Object> The JSON object to be converted
• array: <Array> The array to hold the resulting key:value pairs
• func: <Function> A function to format the key:value pairs

Example:

function buildMetric(n, v) {
    const name = String(n).split(' ').join('_');
    let built = new Metric.setName(name).setValue(v);
    return built;
}

const mongoStat = [];
JSONToKv(info, mongoStat, buildMetric);

dehumanize

dehumanize(string)

Normalise (de-humanize) a value that may have been generatied in "human" readable form. * Takes an input like "10Mb", "5 Kb" or "12.3Kb" and returns a numeric value in bytes

• Returns: <Number
• string <String> The string to convert. e.g: 1 Kb will return 1024

jsonParse

jsonParse(string)

A wrappper for JSON.parse with error handling

• Returns: <Object>
• string: Input JSON as a string

isInFilters

isInFilters(item, filterList)

A filter utility that evaluates against lists of strings or regex

• Returns: <Boolean>
• item <String> The string we're looking for
• filterList <Array> An array of regular expressions or stringto test Example

mongoStat.filter((m) => passFilter(m.name, filters)).forEach((m) => {
    filteredMongoStat.push(m);
});

Where the filter list is:

"filters": [
    "text",
    "/regex\\s/",
    "!/inverseRegex/"
]

passFilter

passFilter(item, flist)

A basic, but very quick exclusion filter

• Returns: <Boolean>
• item <String> The string we're looking for
• flist <Array> An array of regular expressions to test, matching items will be filtered (dropped) Where the filter regex is of the following format:

"filters": [
    "'^tcmalloc'",
    "'foo'"
]

REST

head(url, params, httpOptions)
post(url, params, httpOptions)
get(url, params, httpOptions)

The REST module provides wrappers for the HEAD, POST and GET HTTP(S) requests, and returns the resulting content (if any), or a detailed error. Additionally timing is done at various stages, enabling the requests to be used as a basic synthetic test.

• Returns: <Object>
• url: <String> A URL, containing the protocol
• params: <Object> OR <String> Query parameters, or post data. In most circumstances, the object will be appropriate, but a string can be accepted for instances where the query parameters have to be URI encoded
• httpOptions: <Object> HTTP Options

Return Object:

|Key|Type|Description| |---|----|-----------| | exception | <String> | Any non HTTP error | | | status_code | <String> | HTTP status code (if available) | | body | <String> | HTTP response body | | roundTripTime | <Number> | Round trip from request initiation to full body received | | responseTime | <Number> | HTTP Header received | | errorTime | <Number> | Time to detect / receive fatal error | | dnsTime | <Number> | DNS lookup time from socket library | | connectTime | <Number> | Connect time from socket library | | certValidTo | <String> | Certificate expiry as a string | | certValidToMs | <Number> | Certificate valid time remaining | | headers | <Object> | HTTP headers |

Example:

A simple synthetic test, using the Boolean detector for string validation, and Adaptive detector for Round Trip Time

const target = {
    "url":"http://example.com",
    "host": "example.com",
    "validString": "Example",
};

async function checkUrl(target) {
    const { url } = target;

    const request = async () => {
        const metrics = [];
        const result = await rest.get(url, {});
        let success = result.status_code === 200;

        //
        // If the request returned 200, move on to string validation
        //
        if (success) {
            const page = result.body.toString();
            if (page.match(target.validString) === null) {
                success = false;
                result.exception = `Validation string not found: '${target.validString}'`;
            }
        }

        const {
            roundTripTime,
            responseTime,
            errorTime,
            dnsTime,
            connectTime,
            certValidTo,
            certValidToMs,
        } = result;

        //
        // Build the RTT metric
        //
        const rttMetric = new Metric()
            .setMetric('roundTripTime')
            .setSource(target.host)
            .setData(roundTripTime)
            .setAdditionalData({
                responseTime,
                errorTime,
                dnsTime,
                connectTime,
                certValidTo,
                certValidToMs,
            });
        metrics.push(rttMetric);

        //
        // Build the status metric
        //
        const statusMetric = new Metric()
            .setMetric('urlcheck')
            .setSource(target.host)
            .setData(success)
            .setAdditionalData({
                exception: result.exception,
                status_code: result.status_code,
            });
        metrics.push(statusMetric);
        sendMetrics(metrics);
    };
    request();
}