api-core

The generic component for creating an API with Blueprint support.

This repo is meant to be required as a module on other domain-specific API repos.

• Design decisions
• How do I set this up?
• How do I configure this?
• How do I make requests? (example run)
• How do I add a new endpoint?
• How do I test this?
• How do I make a release?

Design decisions

Authenticaton

Redis is used the state for OAuth2 (tokens, etc...).

The OAuth2 scopes are defined as follows:

• if none is specified, the service is used for internal OAuth2 authentication purposes and, thus, is not exposed through HTTP

• if public the service is available publicly through HTTP (i.e. it does not require any authentication)

• for any other value, the service is available through HTTP, and this property is used for OAuth2 scope authorization purposes

You can overview the decisions behind authentication processes here.

HTTP

The content type of all responses is application/json. For security reasons, only application/json requests are accepted.

• When a resource does not exist:

  • GET a single resource

    • return an HTTP 404 response

  • GET a list of resources

    • return an empty array

  • UPDATE a resource

    • return an HTTP 404 response

Data persistence

PostgreSQL is currently the only supported database.

Services

There was the need to decouple data access operations from business logics for maintainability reasons.

For this reason, there are two kinds of services: DataAccess and Business.

Business services deal with implementing business logics and/or interacting with system components that are external to an API. They may communicate with multiple DataAccessService to achieve this goal (0..*).

DataAccess services aim to fetch data from a database (e.g. PostgreSQL). They may be shared between Business services (1..*).

How do I set this up?

1. Install Node.js and the project's dependencies

   1. get NVM

   2. cd api-core

   3. nvm install

   4. npm install

macOS

2. Install Redis

   • brew install redis

3. install PostgreSQL

   • brew install postgres

Ubuntu 14.04

2. Get ready to use PPAs

   • we're trusting Chris Leas's PPA. He's legit because of Chris Lea Joins Forces With NodeSource

     sudo apt-get -y install python-software-properties
     sudo add-apt-repository -y ppa:chris-lea/redis-server
     sudo apt-get -y update

3. Install Redis

   • we need >= 3.0.x to be safe

   apt-get install redis-server

4. Install PostgreSQL

   apt-get install postgresql-9.4

How do I configure this?

This module's configuration is environment variable-based.

The following env vars are supported:

• TOKEN_SIZE

  • is the size of both access and refresh tokens in bytes

  • e.g. 16 days

• CLIENTS

  • is the comma-separated set of colon-separated client properties to setup on Redis on API load. These can be used for the client_credentials OAuth2 grant type.

  • note that all client scopes must be included on the SCOPES env var

  • client_type:client_id:client_secret:scope -> e.g. confidential:test:test:test days

• SCOPES

  • is the comma-separated list of colon-separated pairs of scopes and their expiration relative to the default ACCESS_TOKEN_DURATION

  • e.g. SCOPES=user:5,retailer:1,scraper:Infinity

• ACCESS_TOKEN_DURATION

  • is the amount of days an access token lasts

  • e.g. 0.04 days ~ 1 hour

• REFRESH_TOKEN_DURATION

  • is the amount of days an refresh token lasts

  • e.g. 7 days

• DBS

  • is the comma-separated list of allowed DBs

  • e.g. DBS=operational,analytics,billing

• ANALYTICS_DB, OPERATIONAL_DB and vars of the form X_DB

  • they are postgres URIs

  • e.g. ANALYTICS_DB=postgres://postgres:@localhost/analytics

  • e.g. OPERATIONAL_DB=postgres://postgres:@localhost/operational

• OPERATIONAL_DB_HOST

  • is the hostname where the operational db is located

  • e.g. localhost

• NODE_DEBUG

  • may be used to expose debug logs for different components

  • e.g. NODE_DEBUG=runner,config,services,postgres,store,api,oauth,web,auth node index.js api.json

How do I run this?

• Pick your poison

  • to start it as a regular Node.js application pick either option, where CONFIG is the path to the apib file:

    • node index.js $CONFIG

    • CONFIG=$CONFIG node index.js

  • to run it with PM2

    1. ensure it's installed

    * `npm install -g pm2`

    2. pm2 start ecosystem.json --env env_name

       • where env_name is the environment to run the API under (e.g. production)

       • --env env_name should be ommited if running on development mode

How do I make requests? (example run)

Note: this repo needs a supermodule to include it as a submodule. This section is meant as an example of steps that should be executed from the said project.

1. Run the API

2. Get an access_token

   • with curl

     • DATA="{ \"username\": \"$EMAIL\", \"password\": \"$PASS\", \"grant_type\": \"password\", \"scope\": \"retailer\", \"client_id\": \"test\", \"client_secret\": \"test\" }"

     • export ACCESS_TOKEN="$(curl -v http://localhost:3000/oauth/token -H "Content-Type: application/json" -X POST -d "$DATA" | JSONStream 'access_token' | tr -d '[]"')"

   • with the api-client

3. check if you got an access_token

   • with curl

     • echo $token

   • with the api-client

4. Make a request using the access_token you just got

   • with curl

     • curl -iv http://localhost:3000/dummy -X GET -H "Authorization: Bearer $token"

       • if everything's fine, you should have gotten a HTTP 200 OK response and some JSON payload

   • with the api-client

How do I add a new endpoint?

1. Declare your new endpoint's behaviour. To do so , you may use:

   • an *.apib API Blueprint file

     Note: Using an .apib file is preferrable, as Dredd can be used to validate its structure.

     • Use JSON SCHEMA to define the various request-reply flows for the endpoint. More info on JSON Schema draft v4.

     • the OAuth scope can be defined on the Transaction section before the service name

       • e.g. retailer:updateRetailerSettings [POST]

       • e.g. public:getDomainProductPrice [GET]

       • Note: to have a service that is used only for internal authenticaton purposes, only the service name should be provided, i.e email [/oauth/token]:

     • the db to use can be defined on the Request section after the string to

       • the respective env var is fetched from this value.

       • e.g. Request to analytics (application/json; charset=utf-8)

     • the data services that will be used by the associated business service can be defined on the Request section after the string with data

       • e.g. Request to analytics with data [service], [service] ... (application/json; charset=utf-8)

   • a configuration file *.json

     • Expose the services through an array of service objects whose options are as follows:

       • name [mandatory]: the name of the service for which to spawn data and business components

       • data [optional]: an array of data access services the business service is able to communicate with

         • default: the value for name

       • route [optional]: the HTTP route to reach the service

         • default: the value for name

         • look into express for more information

       • method [optional]: the HTTP method to reach the service

         • default: GET

       • scope [optional]: a string or boolean to specify the service visibility

         • default: the service is used for internal authentication purposes and is not exposed

       • db: the address for the service's PostgreSQL database or the name of the DB to use

         • e.g. postgres://postgres:@localhost/db_name

         • e.g. operational

2. Setup the data access service

   • Note: your service should inherit from the base service class and override the getSQL method

     • alternatively, if there's a helper class under data/*.js, you might only have to override getSQL

   • create a stub under data/serviceName.js

     const DataAccessService = require('../api-core/lib/data/index');
     
     module.exports = class Service extends DataAccessService {
       getSQL(params) {
         // return the SQL the service is going to execute
         // it should be parameterized with ${parameterName}
         return this.SQL`the sql`;
       }
     }

     • don't forget to remove the comments

3. (Optionally) setup the business service

   • Note: if there's no need to extend the default BusinessService functionality, you shouldn't create any file. A default service will be loaded instead

   • create a stub under business/serviceName.js

     const BusinessService = require('../api-core/lib/business/index');
     
     module.exports = class Service extends BusinessService {
       run(params) {
         // Coordinates how different requests to data access services should be orchestrated.
         // If not overridden, all services described under the `data` config
         // will be called simultaneously and their results aggregated
         // It should return a Promise.
       }
     }

4. Implement your services

   • Fill in the code to make them useful

   • Before committing any external dependency be sure to check the project with NSP and be sure no new vulnerabilities are found.

How do I test this?

npm test

How do I make a release?

1. Install git-flow

2. npm version $SEMVER

3. git flow release start $VERSION to start a new release

4. git flow release finish $VERSION to finish the release

   • develop gets merged to master