Backtrace Service Layer nodejs library

Library of common functions needed for Node.js Backtrace services.

Getting started

$ npm install
$ npm run build
$ npm run test

Then in your code you can include by using

import * as backtrace-service from 'backtrace-service'

or use require instead

const backtraceService = require('backtrace-service');

Service layer integration

To start using Service Layer Integration you have to create new object of a IdentityManager class. After that you can use serviceRequest and authenticateRequest methods.

For an example service implementation, see here.

Service flow

Services are expected to configure one or more authentication tokens, which will be distributed to coronerd instances using the service. While a service is running, it may receive registration requests at ${url_prefix}/service, which enable the service to integrate new coronerd instances on the fly.

Coronerd URLs provided as the url parameter to coronerdCallback are the base URLs, for example, https://backtrace.sp.backtrace.io/. Services may expect to append the appropriate resource for their needs, e.g. /api/config, to reach that resource on the coronerd instance.

What data coronerd will send to service to register? Coronerd via HTTP POST will send in a request body:

• action - action name (for example register),
• url - coroner url (for example https://yolo.sp.backtrace.io/)
• service_to_coronerd_url - where the service should make requests to coronerd (default to url if omitted)
• nonce - auth token
• hmac - Hash Message Authentication Code

Quick Example

TypeScript:

import { ICoronerRequestOption, IdentityManager } from 'backtrace-service';
...

const idm = new IdentityManager(serviceName, serviceSecret);
app.post(
  '/api/{serviceName}/service',
  idm.serviceRequest({
    name: serviceName,
    secret: serviceSecret,
    coronerdCallback: (url: string) => {
      //coroner callback
    },
    logger: logger
  } as ICoronerRequestOption),
);

JavaScript:

const btservice = require('backtrace-service');
const idm = new btservice.IdentityManager(serviceName, serviceSecret);
app.post(
  '${url_prefix}/service',
  idm.serviceRequest({
    name: 'simple',
    secret: 'asdfghjk',
    coronerdCallback: (url) => {
      console.log('heard from coronerd at: ' + url);
    },
  }),
);

Options for serviceRequest

The following options are accepted as the sole argument for the call:

• name: Name of the service (usually its type).
• secret: The shared secret that the service will use to authenticate incoming requests and config replies from a coronerd.
• coronerdCallback: A callback that takes a URL parameter, and performs any service specific setup associated with integrating a new coronerd instance.
• logger (optional): An object that has a log function which can be logged to. For example, winston logger instances.

authenticateRequest usage

This function is intended as an additional middleware which may be used in application routes to validate requests that involve a session token issued by a remote coronerd. The actual call can reuse the same options argument used for serviceRequest, although it does not use coronerdCallback.

In the route middleware list, prior to authenticateRequest, the application must attach a req.coronerAuth object which contains:

• url: The full URL to the remote coronerd instance.
• token: The user's session token to be validated.

For example:

req.coronerAuth = {
  url: "https://backtrace.sp.backtrace.io/",
  token: "f5af46b8eb32adb860ef46a9e714cfde",
}

If req.coronerAuth object is undefined, authenticateRequest method will try to retrieve token and location information from headers. If you prefer to use headers instead of extending request object please set X-Coroner-Token and X-Coroner-Location

This normalized form is used due to the fact that different services take these parameters from clients in different ways.

Middlewares that come after authenticateRequest will have access to the validated coronerd /api/config response in req.coronerAuth.cfg.

Quick Example

Example of sample middleware

TypeScript:

import { ICoronerRequestOption, IdentityManager } from 'backtrace-service';
...

@Middleware({ type: 'before' })
export class AuthMiddleware implements ExpressMiddlewareInterface {
  private readonly _identityManager = new IdentityManager(serviceName, serviceSecret);
  public use(req: express.Request, res: express.Response, next: express.NextFunction): any {
    return this._identityManager.authenticateRequest()(req, res, next);
  }
}

In example above middleware will try to retrieve X-Coroner-Token and X-Coroner-Location from request headers.

JavaScript:

const idm = new btservice.IdentityManager();
// create express app...
// prepare utility method
function prepAuth(req, res, next) {
  const auth = {
    token: req.get('X-Coroner-Token'),
    url: req.get('X-Coroner-Location'),
  };
  req.coronerAuth = auth;
  next();
}

// create application options
const svcOpts = {
  name: 'simple',
  secret: 'asdfghjk',
  coronerdCallback: (url) => {
    log('info', 'heard: ' + url);
  },
  logger: logger,
};

app.post('/api/{name}/{action}', prepAuth, idm.authenticateRequest(svcOpts), (req, res) => {
  res.json(req.coronerAuth);
});

Project Validation

Project Validation supports gating sensitive information in services on a project-basis for Backtrace Teams. The projectValidation middleware ensures all Express response.send invocations procceed successfully only if the projectValidated flag is true. This is intentionally not middleware in order to support the case where the route handler has to do some work before determining projects in scope (e.g. backtrace-sca needs to fetch correlated defects before it is aware of the projects that these defects come from). To set the projectValidation flag to true, two methods are available for use in service route handlers.

• checkProjectAccess

  This is to be used in routes that use the authenticateRequest middleware. It relies on /api/config to validate the request's list of scoped projects. If the scoped list is valid, the res.send will be successful. If invalidated, the service will return a 403. scopedProjects can be a list of project names or pids.

  service.app.get(
  '/api/my-service/use-projects',
  service.identityManager.authenticateRequest(),
  async (req: express.Request, res: express.Response, next: express.NextFunction) => {
    // ... service-related work
    const scopedProjects = ["my-project"]; // Projects needed to fulfill request.
    service.identityManager.checkProjectAccess(req, res, scopedProjects);
    res.send({ message: "Expected result." });
  }
);

• bypassProjectAccessCheck

  For service route handler's that do not care about project access at all, you can intentionally bypass the project validation step.

  service.app.get(
  '/api/my-service/get-version',
  async (req: express.Request, res: express.Response, next: express.NextFunction) => {
    // ... service-related work
    service.identityManager.bypassProjectAccessCheck(res);
    res.send({ message: "Expected result." });
  }
);

Utils

Backtrace-Service tools offers few utility methods. By using them we want to standarize how we read application configurations/log application messages/create custom auth methods.

Identity Manager

Service Layer Integration using IdentityManager class to provide integrations methods with coronerd. If you want to write auth method to coronerd or check if token is valid you can use methods available in aIdentityManager instance.

Quick Example:

import { IdentityManager } from 'backtrace-service';
const token = //auth token
const universeUrl = //universe url: https://yolo.sp.backtrace.io
const identityManager = new IdentityManager(serviceName, serviceSecret);
// get configuration from coronerd
const configuration = await identityManager.getConfiguration(universeUrl, token);

// validate token
const validToken = await identityManager.isValidToken(universeUrl, token)

loginCoronerd

A service may login to a coronerd via its base URL using this method. This performs a login using the service HMAC exchange, returning a configuration that includes a token usable for service purposes.

const cfg = await identityManager.loginCoronerd("https://foo.sp.backtrace.io");

GetConfiguration

IdentityManager method that allows you to prepare a request to coronerd to retrieve configuration. Please keep in mind, if you're using SL integration, configuration should be available in request.coronerAuth.cfg. GetConfiguration method using Promise API to get data from a server.

IsValidToken

IdentityManager method that allows you to validate token with coronerd. In this case IdentityManager will return boolean information - true if coronerd returns valid configuration, otherwise false. IsValidToken method using Promise API. Please keep in mind to use await or then/catch syntax to get a result from method.

Configuration

Backtrace-Service package allows you to read service configuration file. To standarize paths used in your service to configuration files we suggest to use these methods to easily integrate service with other services.

GetDescriptor

A service can obtain its service integration settings by using getDescriptor, which returns an IServiceDescriptor. It must specify a default port number, to be used in on-premise configurations. It may specify additional default parameters as IDescriptorOpts.

The values returned in the IServiceDescriptor must be used by the service in order to properly integrate. This function generates a descriptor file, if necessary, otherwise it uses what's provided.

const descriptor = getDescriptor('service-name', 12345);

listenDescriptor

This is a helper function that reduces boilerplate by allowing services to listen on the descriptor-specified port in the descriptor-specified manner.

The service must provide its SSL credentials (if it has any) as well as the app to process requests. It is valid for no SSL credentials to pass, although in this case, only descriptors with protocol 'http' will work.

// fill in from config file, parameter is as used by https.createServer().
const credentials = { key: "", cert: "" };
listenDescriptor(descr, app, credentials, () => {
  console.log(`Listening on ${descr.port}`);
});

GetConfig

getConfig fetches configuration from the expected place on the machine (outside the service). If configuration doesn't exists then getConfig method will try to fetch configuration from internal service path Algorithm:

1. Try to fetch configuration from: /etc/backtrace/${serviceName}/${serviceName}.conf
2. If configuration from #1 doesn't exists try to read configuration frile from path: ${process.cwd()}/${serviceName}.conf
3. Return config or undefined

example:

import { getBackupConfig, getProperConfig, getConfig } from 'backtrace-service';

//read backup configuration - service internal configuration
const serviceConf = getBackupConfig('service-name');
//read production configuration - outside service
const prodConf = getProperConfig('service-name');
//get configuration - depends on preferences
const conf = getConfig('service-name');

UniverseHelper

Utility methods that allows you to get more information about Backtrace universe from a url.

getBacktraceHostName

GetBacktraceHostName method allows you to get Backtrace host name from provided urlString. For example - method will return https://yolo.sp.backtrace.io from url https://yolo.sp.backtrace.io/blah/poop/doop?query=123&anotherQuery=nbgaubakgb

getBacktaceUniverseName

GetBacktaceUniverseName method allows you to retrieve a Backtrace universe name. For example: method will return yolo from url https://yolo.sp.backtrace.io or https://submit.backtrace.io/yolo/....

example:

import { UniverseHelper } from 'backtrace-service';

const url = 'https://yolo.sp.backtrace.io/blah?a=1&b=bgj';
const url2 = 'https://submit.backtrace.io/bolo/token/json';

// method below will print a string 'https://yolo.sp.backtrace.io
console.log(UniverseHelper.getBacktraceHostName(url));

//method below will print a string 'bolo
console.log(UnvierseHelper.getBacktraceUniverseName(url2));

Backtrace service API

Backtrace-service contains a lot useful methods that you can use to generate your express application with all Backtrace best practices. By default backtrace-service package allows you to:

• setup auth,
• setup service layer configuration,
• setup metrics,
• setup integration endpoints,
• setup Backtrace (logger),
• read configuration files (app conf + service descriptor),
• and even more...

We recommend to use BacktraceService.bootstrap method to generate Backtrace api, but if you want to call each methods separately, feel free to use BacktraceService static methods.

Quick sample:

const backtraceService = BacktraceService.bootstrap<IConfiguration>({
  name: 'service-name',
  port: 9999,
});
backtraceService.start();

In TypeScript you can use generic attribute to provide configuration class. In JavaScript, you don't need to provide any generic attribute!

Configuration

BacktraceService in application root directory will read configuration file. By doing that bootstrap method or BacktraceService constructor will read configuration and set configuration data in BacktraceService configuration variable.

By default BacktraceService will read from configuration file:

• metrics object,
• ssl object,
• logger object,
• backtrace object,

In addition to that BacktraceService will extend default configuration with values:

• isProduction - determine environment type,
• descriptor - service descriptor,

Example configuration file:

{
    "api": {
        "prefix": "/api/symbold"
    },
    "db": {
        "type": "sqlite",
        "port": 5432,
        "database": "./symbol-server.db"
    },
    "log": {
        "level": "debug",
        "output": "logs",

    },
    "metrics": {
        "enabled": false,
        "secret": "fake secret",
        "uuid": "fake uuid"
    },
    "backtrace": {
        "enabled": true,
        "backtraceUrl": "https://submit.backtrace.io/sample/99999fb147eccb0c1aa91f03499c96238ceaa3199dbb5cbdbb0a8ed57ae37191/json"
    },
    "ssl": {
      "key": "path/to/key",
      "cert": "path/to/cert"
    }
}

In example above you can see api and db object that will be readed by BacktraceService, but default method won't use these objects to generate your application.

Configuration file allows you to define remote metrics, that backtrace-service will send to Circonus/Backtrace. In case if you would like to disable them, you can use enabled property. If you will set this property to false, backtrace-service will ignore these properties and won't start Backtrace/Circonus integration.

Service Layer

BacktraceService.setupServiceLayer method will generate default service layer configuration for your application. By default this method will read from service/use service descriptor, setup identityManager and create application endpoint.

Usage:

const app = express();
BacktraceService.setupServiceLayer(app, descriptor);

Setup auth

Backtrace.setupAuth method add all security middlewares to your express application - helmet, cors.

Usage:

const app = express();
BacktraceService.setupAuth(app)

Setup middlewares

Backtrace.setupMiddlewares method will add all used middlewares/utilities to your express application - json, compression, pingdom.

Usage:

const app = express();
BacktraceService.setupMiddlewares(app);

Read configuration file

BacktraceService.readConfiguration method allows you to read service configuration files. In addition to that (only in TypeScript), this method allows you to use strongly defined configuration file. By using generic attribute you can pass to readConfiguration method, expected configuration definition. If you want to ues default configuration object or you're JavaScript user, you can ignore generic attribute.

Usage:

const configuraiton = BacktraceService.readCofniguration('backtrace-service-name', 9999);

Bootstrap

All methods above will be used by bootstrap method to extend your express application. Usage:

const backtraceService = BacktraceService.bootstrap<IConfiguration>({
  name: 'service-name',
  port: 9999,
});

Start

You can start express applicaiton by using BacktraceService object. By doing that BacktraceService will use or not use ssl certs, descriptor port and more.

backtraceService.start();

MetricsStorage

BacktraceService constructor will create new MetricsStorage instance class. By doing that our application can send data to circonus. If you bootstrap your application by using bootstrap method you can access metricsStorage by retrieving backtraceService.metrics. If you don't want to use bootstrap method, you can still create MetricsStorage object by using MetricsStorage construtor.

const metrics = new MetricsStorage({uuid: 'uuid', secret: 'secret'});

In addition to that, you can define sending interval. By doing that backtrace-service will send data to Circonus by using your time interval, instead of default value (1000).

If you pass to MetricsStorage construtor express application, metricsStorage will add request latency middleware to express application. By doing that MetricsStorage will send execution time of each action to Circonus.

If you want to get more details from metricsStorage object, you can use logger object to provide default logger methods.

Full sample:

const app = express();
const metrics = new MetricsStorage({uuid: 'uuid', secret: 'secret', interval: 10000}, app, logger);