Prometheus

Background

Shopee provides a complete monitoring and alarm system based Prometheus. So this packages is dedicate to provide a commom SDK that can help all the node server conveniently integrate it. This package can used in all the node server. If you are using some framework you can choose the specific package

• @infra-node-kit/prometheus-nest
• @infra-node-kit/prometheus-koa
• @infra-node-kit/prometheus-express

Initial the metrics

// definition
// this method will call the prom-client collectDefaultMetrics and register the customMetrics
function initMetrics(options?: DefaultMetricsCollectorConfiguration): void;

import { initMetrics } from '@infra-node-kit/prometheus-common'
initMetrics()

Custom registry

import { Registry, initMetrics } from '@infra-node-kit/prometheus-common'
const customRegistry = new Registry()
initMetrics({register: customRegistry})

Collect metrics

you need collect your metrics and return the result. collectRequestMetrics can help you collect the request metrics.

interface ICollectOptions {
  metricsPath?: string
  method?: string
  recordPath?: string
  addRequestLabels?: Record<string, string>
}

/**
 *@description collect the request metrics
 * @param request is the original http request object
 * @param response  is the original http response object
 * @param options.metricsPath is the path of the metrics, api concurrency will skip the metrcis path
 * @param options.method custom the method, default is  request.method.toUpperCase()
 * @param options.recordPath custom the path, default is parseUrlNumberToStar(request.url?.split('?')[0])
 * @param options.addRequestLabels add custom the `requestHistogram` labels
 */
function collectRequestMetrics(request: IncomingMessage, response: ServerResponse, options?: ICollectOptions) {}

Collect example

import {collectRequestMetrics, register} from '@infra-node-kit/prometheus'
import http from 'http'

http.createServer(function(req, res) {
  // do the collect
  collectRequestMetrics(req, res);
  if (req.url?.includes('metrics')) {
    // return the result
    register.metrics().then((data) => {
      res.end(data)
    })
  } else {
    res.end()
  }
}).listen(8080)

collectRequestMetrics custom options

import {collectRequestMetrics, getRegistry} from '@infra-node-kit/prometheus-commom'
import http from 'http'
const metricsPath = '/custom_metrics'
http.createServer(function(req, res) {
  collectRequestMetrics(req, res, {
    metricsPath,
    method:req.method
    recordPath: req.url
  });

  if(req.url === metricsPath) {
    getRegistry().metrics().then(data=>{
      res.end(data)
    })
  }
}).listen(8080)

Custom Metrics

How to get it

import { getCustomMetrics } from '@infra-node-kit/prometheus-commom'
const metrics: ICustomMetrics = getCustomMetrics()
// definition
interface IMaxConcurrencyGauge {
  gauge: Gauge<string>
  inc: (value?: number) => void
  dec: (value?: number) => void
}
interface ICustomMetrics {
  apiConcurrency: IMaxConcurrencyGauge
  outHttpConcurrency: IMaxConcurrencyGauge
  requestHistogram: Histogram<string>
  outRequestHistogram: Histogram<string>
}

Inner Metrics

requestHistogram

new Histogram({
  name: 'http_request_seconds',
  help: 'http request seconds',
  labelNames: ['method', 'path', 'status'],
  buckets: [0.5, 1, 3, 5, 10],
})

It is used to monitor the server http request. It could collect the API QPS、API Cost. collectRequestMetrics include this metric.

http_request_seconds path label all normalized by the internal functions getUrlPurePath, replaceNumberInURLWithAsterisk.

• getUrlPurePath is used to delete the query.
• replaceNumberInURLWithAsterisk is used to replace the number part in the url to a certain value *

apiConcurrency

new Histogram({
  name: 'api_concurrency',
  help: 'api concurrency',
  collect() {
    ...
  }
})

It is used to indicate the number of simultaneous requests being processed.collectRequestMetrics include this metric.

outRequestHistogram

new Histogram({
  name: 'out_request_seconds',
  help: 'out request seconds',
  labelNames: ['method', 'path', 'status'],
  buckets: [0.5, 1, 3, 5, 10],
})

It is used to monitor outbound requests from the current service. It needs you collect yourself.

outHttpConcurrency

new Histogram({
  name: 'out_http_concurrency',
  help: 'out http concurrency',
  collect() {
    ...
  }
})

It is used to monitor the number of concurrent requests made by the server.

custom requestHistogram labels

The requestHistogram is called by the collectRequestMetrics method, so If you want to add the custom label. you can get the requestHistogram and add the label to labelNames and then deliver parameter addRequestLabels to the collectRequestMetrics.

import { getCustomMetrics, collectRequestMetrics } from '@infra-node-kit/prometheus-commom'
let { requestHistogram } = getCustomMetrics()
requestHistogram.labelNames = [...requestHistogram.labelNames, 'customLabel']
http.createServer(function(req, res) {
  collectRequestMetrics(req, res, {
    metricsPath,
    method:req.method
    recordPath: req.url,
    addRequestLabels: {customLabel: 'customValue'} // addRequestLabels
  });

  if(req.url === metricsPath) {
    getRegistry().metrics().then(data=>{
      res.end(data)
    })
  }
}).listen(8080)

Common knowledge

• All the framework, the default metrics path is /metrics
• Use the prom-client default globalRegistry

How to integrate in-house monitor platform

How to set the services monitor and alert