X2 Framework for Node.js | RESTful Web Services

This is X2 Farmework's module that provides foundation for building server-sive applications that expose RESTful APIs. The main purpose of the module is to provide the core that sits between the basic Node.js HTTP server functionality and custom application code responsible for processing the web service API calls. The core implements the most common web service functionality and maps incoming HTTP requests to the application-supplied handlers. It includes hooks for request authenticators and authorizers, CORS, support for multipart responses with streaming data, extraction of parameters from request URIs, marshalling and unmarshalling data, responding to the HTTP OPTIONS method requests, and other web service basics.

See module's API Reference Documentation.

This is a low-level module. For higher level web service building functionality see x2node-ws-resources module.

Table of Contents

• Usage
• Application Configuration
• Endpoints
  • Service Call
  • Service Response
  • Call Authorization
  • Content Negotiation
  • The OPTIONS Method
• Authenticators
  • Actors Registry
    • Caching Actors Registry
  • Basic Authenticator
  • JWT Authenticator
• Authorizers
• Marshallers
• Terminating Application

Usage

The web service is represented by an Application object created by the module's createApplication() function. The application is configured by adding custom endpoint handlers mapped to request URIs using regular expressions. In addition to the endpoints, the application can also include mappings for request authenticators and authorizers. Here is a simple example:

const ws = require('x2node-ws');

ws.createApplication()
    .addEndpoint('/sayhello', {
        GET() {
            return {
                message: "Well Hallo to you!"
            };
        }
    })
    .addEndpoint('/saygoodbye', {
        GET() {
            return {
                message: "OK, bye bye!"
            };
        }
    })
    .run(3001);

This little app will listen on the HTTP port 3001 and will response with a simple JSON object to HTTP GET request sent to either of the two endpoints. For example, for request:

GET /sayhello HTTP/1.1
Host: localhost:3001
Accept: application/json

the response will be:

HTTP/1.1 200 OK
Vary: Origin
Cache-Control: no-cache
Expires: 0
Pragma: no-cache
Content-Type: application/json
Content-Length: 32
Date: Mon, 08 May 2017 21:53:21 GMT
Connection: keep-alive

{
    "message": "Well Hallo to you!"
}

And for an invalid URI request:

GET /invalid HTTP/1.1
Host: localhost:3001
Accept: application/json

it will be:

HTTP/1.1 404 Not Found
Vary: Origin
Cache-Control: no-cache
Expires: 0
Pragma: no-cache
Content-Type: application/json
Content-Length: 74
Date: Mon, 08 May 2017 21:54:30 GMT
Connection: keep-alive

{
    "errorCode": "X2-404-1",
    "errorMessage": "No service endpoint at this URI."
}

The module uses X2_APP section for debug logging. Add it to NODE_DEBUG environment variable to see the debug messages (see Node.js API docs for details).

Application Configuration

The module's createApplication() function that builds and returns the runnable Application object can optionally take an object with the application options. The options include:

• apiVersion - Version of the API exposed by the application. If not specified, the version is determined automatically using the following logic: The NODE_ENV environment variable is examined. If its value is "development", the version is set to the current timestamp so that it changes each time the application is restarted. Otherwise, the version is read from the main module's package.json.

• connectionIdleTimeout - Timeout in milliseconds for inactivity on the HTTP connection when activity is expected from the client (waiting for the request, reading the request body, accepting the server response). If the timeout occurs before the server starts sending the response, an HTTP 408 (Request Timeout) response is sent back to the client and the connection is closed. If timeout happens after the response headers have been sent, the connection is quitely closed. The default is 30 seconds.

• maxRequestHeadersCount - Maximum allowed number of incoming HTTP request headers. The default is 50. This corresponds to the Node.js HTTP module's maxHeadersCount parameter.

• maxRequestSize - Maximum allowed size of request payload in bytes. If exceeded, an HTTP 413 (Payload Too Large) response is send back to the client. The default is 2048.

• allowedOrigins - This is used to configure the CORS. The option is a list (comma-separated string or an array) of allowed CORS origins (e.g. http://www.example.com, etc.). If the front-end application that calls the web service is only available from certain specific URLs, it is recommended to configure the CORS to make certain types of attacks, such as CSRF, harder. If not provided, the default is to allow any origin.

• corsPreflightMaxAge - Part of CORS configuration, maximum age in seconds for caching CORS preflight responses on the client (see "Access-Control-Max-Age" HTTP response header). The default is 20 days.

• delay - Number of milliseconds, by which to delay every response. Can be used in application development to emulate slow backend.

The options object is also made available to the application via the ServiceCall object's appOptions property (described below), so it can be used for custom application options as well.

Once the Application object is created, the following methods are used to configure the web service:

• addEndpoint(uriPattern, handler) - Define the API endpoint. The method associates the application-supplied handler with request URIs that match the uriPattern, which is a regular expression (as a string!). The regular expression is applied to the whole URI, so there is no need to use ^ and $ pattern characters. Also, the expression may contain capturing groups, which become positional URI parameters extracted by the framework from the request URI and provided to the handler when it is invoked. The uriPattern can also be an array, in which case the first element is the pattern and the rest are names for the corresponding positional parameters. The endpoints are matched in the order they were added to the Application object, which prevents ambiguity in the endpoint selection logic when URI patterns overlap. The detailed discussion of the application-supplied handlers is provided in the Endpoints section.

• addAuthenticator(uriPattern, authenticator) - Associate an authenticator with the request URI pattern. Authenticators are responsible for associating actors (see x2node-common module) with requests and are described in detail in the Authenticators section. As with the endpoints, the authenticators are matched in the order they are added (however, normally, an application would have only a single authenticator covering all the endpoints using uriPattern like "/.*"). If no authenticators are added to the application, all requests are processed as unauthenticated (i.e. anonymous).

• addAuthorizer(uriPattern, authorizer) - Associate an authorizer with the request URI pattern. Authorizers are responsible for making the decision whether the authenticated actor is allowed to perform the request or not. It is described in detail in the Authorizers section. The authorizer argument can be a function, in which case it is used as the authorizer's isAllowed() method. As opposed to the endpoints and authenticators, multiple authorizers can match the same URI and they are all called in a sequence rather than only one of them. They are called in the order they were added to the Application object. If no authorizers are added to the application, all requests are passed to the endpoint handlers without any pre-authorization.

• setPrefix(prefix) - Set prefix to be added to the URI patterns for any subsequent addEndpoint(), addAuthenticator() or addAuthorizer() call. Initially empty string.

• addMarshaller(contentTypePattern, marshaller) - Associate a marshaller implementation with a request/response content type. Marshallers are responsible for converting HTTP request and response entities to and from JavaScript objects. The contentTypePattern regular expression (must be supplied as a string!) is matched against the content type as a whole and in a case-insensitive mode. The content type is used without any parameters (such as charset, etc.). Patterns are matched in the order they were added to the application and the first one matched is used. After the application adds (or doesn't add) all of its custom marshallers, the framework automatically adds a default implemention of the JSON marshaller and associats it with content types "application/json" and anything with a "+json" suffix (see RFC 6839). If a request is received with payload and "Content-Type" header, for which the application does not have a marshaller, it responds with an HTTP 415 (Unsupported Media Type) response. See Marshallers section for information on how to add custom marshallers for other content types.

Once the Application object is completely configured, it can be started using its run() method. As its first argument, the method takes the HTTP port, on which the application will be listening for the incoming requests. As an optional second argument the method can take a shutdown hook function, which is called right after the HTTP server is asked to shut down (and stops accepting new connections) but before it completes shutting down. The method ultimately ends up calling standard Node.js HTTP server listen() method. The method returns the instance of Node.js HTTP server it created.

Endpoints

The web service API is represented by the endpoints. An API endpoint is a specific HTTP request URI pattern and a collection of HTTP request methods that can be sent to it. The endpoint call processing logic is implemented in the endpoint handler. The handlers are where the most of the application logic is coded.

Handlers are associated with URI patterns to create API endpoints using Application object's addEndpoint(uriPattern, handler) method. For every HTTP method that the handler supports, it has a method with the HTTP method's name, all caps (e.g. GET(), POST(), etc.). If a request is sent to the endpoint using an HTTP method not supported by the handler, an HTTP 405 (Method Not Allowed) response is sent back to the client. Otherwise, the corresponding method on the handler is called and it return value is used to create the HTTP response.

Service Call

The methods on the handler receive a ServiceCall object as its only argument. The ServiceCall exposes the following properties:

• id - A string representing the unique id of the service call. The id is unique within the service process and can be used to identify requests.

• timestamp - The timestamp when the call was received. Date.now() is used to get the timestamp.

• apiVersion - Application API version (see apiVersion application configuration option).

• appOptions - Application configuration options originally passed to the module's createApplication() function, or an empty object if none were passed.

• httpRequest - The original Node.js http.IncomingMessage representing the HTTP request.

• method - The request method, all caps. This is a shortcut for httpRequest.method.

• requestUrl - Fully parsed request URL represented by a Node.js Url object. The query string is parsed.

• authenticator - The authenticator used to authenticate the request, if any. This is the authenticator added to the Application via its addAuthenticator() method and matched against the request URI.

• authorizers - Array of authorizers used for the call, if any. These are the authorizers added to the Application via its addAuthorizer() method.

• handler - The handler (which is also going to be this in the handler method call).

• uriParams - If the URI pattern passed to the addEndpoint() method has capturing groups, the extracted from the request URI group values are stored in this string array and passed to the handler. If names were provided for the parameters (the addEndpoint() call used an array for the URI pattern and the positional parameter names), the parameters are also available on the uriParams as object properties.

• actor - The actor associated with the call, or null if unauthenticated. Note, that this is a read-write property. If the handler sets a new actor to the ServiceCall object, the authenticator may pick it up and adjust the response accordingly.

• authorized - A Boolean flag that tells if the call was authorized. By the time the call object is passed to the handler, the flag is going to be true.

• requestedRepresentation - Response content type requested by the caller via an HTTP request "Accept" header. If the caller did not provide "Accept" header, defaults to the first content type returned by the handler's optional getRepresentations() method. If the handler does not have getRepresentations() method, defaults to "application/json".

• entity - An object with the unmarshalled request payload, or null if none.

• entityContentType - If entity is present, this is the request payload content type (all lower-case, stripped of any parameters such as "charset").

Service Response

Upon completion, the handler method may return one of the following:

• A null, in which case an HTTP 204 (No Content) response is sent back to the client.

• An object, in which case it is serialized into JSON and is sent back to the client in an HTTP 200 (OK) response payload.

• A ServiceResponse object created via the module's createResponse() function described below.

• Anything else, in which case it is converted into a string and is sent back to the client as "text/plain" content type in an HTTP 200 (OK) response payload.

• A Promise of anything of the above. If the promise is rejected with a ServiceResponse object, that's the response that gets sent back to the client. If it is rejected with anything else, an HTTP 500 (Internal Server Error) is send back to the client.

The most specific way of creating a service call response is by using the module's createResponse() function. The function takes a single argument with the HTTP response status code. The ServiceResponse object that it returns exposes the following properties and methods:

• setHeader(name, value) - Add header to the HTTP response. Any previously set header is replaced. The header name is identified by the name argument and is case-insensitive. If the value is an instance of Date, it is automatically formatted (using Date.toUTCString()).

• addToHeadersListHeader(name, value) - Add value(s) to an HTTP response header that is a list of other header names. Examples of such headers are "Vary", "Access-Control-Allow-Headers" and "Access-Control-Expose-Headers". The method checks if the headers are already present in the current value and does not add them twice. The header name is specified by the name argument and is case-insensitive. The value can be a string or an array of strings. The case of the header names in the value is also case-insensitive (automatically normalized by the method).

• addToMethodsListHeader(name, value) - Add value(s) to an HTTP response header that is a list of HTTP methods. Examples of such headers are "Allow" and "Access-Control-Allow-Methods". The method checks if the methods are already present in the current value and does not add them twice. The header name is specified by the name argument and is case-insensitive. The value can be a string or an array of strings. The case of the method names in the value is also case-insensitive (automatically normalized by the method).

• setEntity(data, [contentType]) - Add main entity to the HTTP response (any previously set entity is replaced). The entity data specified by data argument can be one of the following:

  • An object, in which case it is serialized using a marshaller associated with the specified contentType. Custom marshallers can be registered on the Application via its addMarshaller() method.

  • A Node.js Buffer, in which case the buffer's binary data is sent in the response body.

  • A Node.js stream.Readable. If used, the response is sent using "chunked" transfer encoding (see HTTP specification's Chunked Transfer Coding).

  If the contentType argument is not provided, "application/json" is assumed.

• addAttachment(data, [contentType], [filename]) - Add attachment to the HTTP response sent back to the client in the response payload. The attachments are semantically different from the main response entity set by the setEntity() method. Multiple attachments can be added to the response with or without the main entity (although normally with). If a response ends up having an entity and attachments or just more than a single attachment, the HTTP response is sent with content type "multipart/mixed". The parts are included in the response payload in the order they were added with the main entity, if any, always first. As with the setEntity() method, the data can be an object, a buffer or a stream. The default contentType is "application/json" and providing a filename argument will include "Content-Disposition" response header with the specified "filename" parameter.

• statusCode - The HTTP response status code.

• hasHeader(name) - Returns true if the specified response header is present on the response. The name argument is case-insensitive.

• headers - HTTP response headers present on the response. The property is an object with keys being all lower case header names and the values being strings with the corresponding header values.

• entities - An array of the HTTP response entity and the attachments in the correct order, or an empty array if none. Each array element is an object that includes a headers property (same format as the headers property of the response object) and a data property, which is an object, a buffer or a stream.

All of the response construction addXXX() and setXXX() methods return the response object itself for chaining.

The x2node-ws module, in addition to the createResponse() function, also exposes a function called isResponse(). It takes an object as its single argument and returns true if the provided object is a ServiceResponse.

Call Authorization

An enpoint handler can provide an optional method called isAllowed(), which is called by the framework before any service call is forwarded to the main processing method to give the handler an early chance to check if the actor associated with the call is allowed to perform it. The method, if defined, receives the ServiceCall object as its only argument with the actor property set. The method returns a Boolean or a Promise of it. If it is true, the call is forwarded to the endpoint handler's main call processing method. If it is false, the call is aborted and the client gets either an HTTP 401 (Unauthorized) response if the request is not authenticated (actor property on the call is null) or an HTTP 403 (Forbidden) response if it is.

Content Negotiation

An endpoint handler can provide an optional method called getRepresentations(). The method is called by the framework if the incoming request has "Accept" header. The method takes the ServiceCall object as its only argument and returns an array of content types that it can generate. If handler does not have getRepresentations() method, it is assumed that it only generates responses in "application/json" format.

The OPTIONS Method

The OPTIONS handler method, if present, is special. The framework takes care of responding to the OPTIONS requests on its own, but before sending the response it can give the handler a chance to participate in building the response if it defines an OPTIONS method. As opposed to the normal HTTP method handler methods, the OPTIONS method receives two arguments: the ServiceCall and the ServiceResponse. It can add headers to the provided ServiceResponse, if it needs to, and the return value of the method is ignored.

Authenticators

Before the call is passed to the matching endpoint handler, it is passed to an authenticator addded to the Application using its addAuthenticator() method. The authenticator is responsible to identifying the actor making the call and setting it to the ServiceCall.actor property. The authenticator has the following interface:

• authenticate(call) - Method called by the framework to authenticate the call. The call argument is an instance of ServiceCall. The method returns an actor object, a null if the call cannot be authenticated, or a Promise of the above.

• addResponseHeaders(call, response) - An optional method that an authenticator can have if it needs to add headers to the HTTP response. The method is called whenever the framework is sending an HTTP response after the call has been passed through the authenticate() method. The call argument is an instance of ServiceCall and the response argument is an instance of ServiceResponse.

Actors Registry

The task of request authentication has two distinctive parts: extracting the authentication information such as the caller handle and credentials from the request (e.g. from the HTTP request headers) and then looking up the actor in some sort of a user database. To decouple the task of the actor lookup from the authenticator the framework introduces an ActorsRegistry interface. The interface includes one single method:

• lookupActor(handle, [creds]) - Lookup the actor in the actor registry. The actor is identified by the string argument handle, which is the actor handle (user id, login name, etc.) extracted by the authenticator from the request. Optionally, if the authentication scheme and the actors registry include it, the second string argument creds is the actor credentials (for example the password) also extracted by the authenticator from the request. The method returns an Actor object or a Promise of it. It it is null, or if the returned promise is fulfilled with null, no valid actor with the specified handle and credentials exists. If it is a Promise and it is rejected, it is considered an unexpected internal error, which normally results in an HTTP 500 response.

The authenticators do not have to use the actor registries, but it is a recommended practice.

Caching Actors Registry

For convenience, the module provides a class CachingActorsRegistry. The class wraps another actors registry and caches the lookup results for a TTL period of time in memory. For example:

const ws = require('x2node-ws');

// let's say that our app specific actors registry is implemented as a class
const MyActorsRegistry = require('./lib/actors-registry.js');

// wrap an instance of our app actors registry in a cache
const actorsRegistry = new ws.CachingActorsRegistry(
    new MyActorsRegistry(), 100, 10000);

The caching actors registry constructor takes three arguments:

• registry - The underlying actors registry.
• maxCached - Maximum number of unexpired cached actors to allow to keep in the memory. In the example above, we allow up to 100 actors to be cached. If the capacity is exceeded, the underlying registry is called directly and the result of the call is returned without caching. At the same time, an error message is logged recommending the cache capacity increase.
• ttl - Cached actor TTL in milliseconds. The actor is reloaded from the underlying registry after this amount of time. In the example above we keep cached actors in memory for 10 seconds (10000 milliseconds), which is useful for increasing the performance of clients that send sequences of API calls to perform this or that operation.

The caching registry instance exposes the following methods:

• invalidateCachedActor(handle) - If actor with the specified handle is cached, invalidate the cache entry. This method is useful when the application changes something about the actor, for example the actor permissions, and wants the authenticators to start using the updated actor immediately instead of waiting for the cached actor TTL expiration.

Basic Authenticator

The x2node-ws module includes an authenticator implementation for the "Basic" scheme (see RFC 7617). The authenticator class is exported by the module as BasicAuthenticator. The constructor takes two arguments: the actors registry (an implementation of the ActorsRegistry interface) and an optional authentication realm with the default value of "Web Service". For example:

const ws = require('x2node-ws');

ws.createApplication()
    .addAuthenticator('/.*', new ws.BasicAuthenticator({
        lookupActor(loginName, password) {
            if ((loginName === 'myuser') && (password === 'mypassword'))
                return Promise.resolve({
                    stamp: 'myuser',
                    hasRole: () => true
                });
            return Promise.resolve(null);
        }
    }, 'My Service'))
    // configure the rest of the web service
    // ...
    .run(3001);

Note the use of a dummy actors registry implementation. Such implementations are often useful for testing and development environments.

JWT Authenticator

A JWT-based authenticator implementation (for OAuth 2.0, etc.) is provided by the x2node-ws-auth-jwt module.

Authorizers

An individual endpoint handler can have an isAllowed() method where it makes the decision if the authenticated actor is authorized to make the call or not. However, often the same call authorization logic is applied across a whole bunch of endpoints. Instead of replicating the same logic in every handler, the application can register an Authorizer for a URI pattern that covers all the protected endpoints using the Application object's addAuthorizer() method. The first argument of the method is the URI pattern and the second argument is an implementation of the Authorizer interface, which includes a single isAllowed() method defined the same way as the one on the endpoint handler:

• isAllowed(call) - Tell if the specified by the call argument ServiceCall is allowed to be performed by the actor in the ServiceCall.actor property. The method can return a Boolean or a Promise of it. If it is true, the call is forwarded further to other matching authorizers and utlimately to the endpoint handler. If it is false, no other authorizers are called, no handler is called, and the client gets either an HTTP 401 (Unauthorized) response if the request is not authenticated or an HTTP 403 (Forbidden) response if it is.

The second argument of the addAuthorizer() method can be also a function, in which case it used as the authorizer's isAllowed() method.

As opposed to the authenticators and endpoint handlers, multiple authorizers can be matched against a request URI. If so, they are called in a sequence in the same order as they were added to the Application object. If the handler also has an isAllowed() method, it is called last. Only if all the authorizers in the chain and the handler's isAllowed() method, if any, tell that the call is allowed, the call is forwarded further to the endpoint handler's main call processing method.

Marshallers

Marshallers are used to deserialize (unmarshal) HTTP request entities into JavaScript objects and to serialize (marshal) JavaScript objects into HTTP response entities. Marshallers are associated with content types (values of the "Content-Type" HTTP header). By default, the Application includes a JSON marshaller associated with "application/json" conent type and anything with a "+json" suffix. A custom marshaller can be added to the Application object using its addMarshaller() method. The method takes two arguments: the content type regular expression pattern and an implementation of the Marshaller interface, which includes two methods:

• serialize(obj, contentType) - Serialize the specified by the obj argument object into the binary data for the specified contentType. The contentType argument may include an optional "charset" parameter. The method returns a Node.js Buffer object with the serialized data.

• deserialize(data, contentType) - Deserialize the Buffer provided as the data argument into a JavaScript object using the specified contentType. The contentType argument may include an optional "charset" parameter. The method returns the deserialized object. If the binary data in the buffer is invalid and cannot be deserialized, the method must throw an X2DataError (see x2node-common module), which will result in an HTTP 400 response. Alternatively, the method may throw a service response object.

An individual handler can provide custom deserialization functions for specific content types in addition to the marshallers registered on the application. The handler can include a property named requestEntityParsers, which is an object with keys for the content-types and values providing the deserialization function for that content type. The deserialization function follows the signature of the marshaller interface's deserialize() method. Any deserializer provided by a handler superceeds the marshaller on the application.

The module provides a simple deserializer function exported as TEXT_DESERIALIZER. All it does is it converts the binary data in the request entity to a string and returns an object with one property called text, which contains the text. For example:

ws.createApplication()
    .addEndpoint('/text', {
        requestEntityParsers: {
            'text/plain': ws.TEXT_DESERIALIZER
        },
        POST(call) {
            console.log('ENTITY:', call.entity);
        }
    })
    .run(3001);

Note, that TEXT_DESERIALIZER provided by the module only supports the following charsets: US-ASCII, ISO-8859-1, UTF-8 (the default) and UTF-16LE.

Terminating Application

Once the Application object's run() method is called, Node.js process will keep running and listening to the incoming requests on the specified TCP port. To stop the web service application, either of the following signals can be sent to it: SIGHUP, SIGINT (the Ctrl+C), SIGTERM (standard system signal used to terminate background processes) or SIGBREAK (Ctrl+Break on Windows).

The Application object is an EventEmitter, which emits a "shutdown" signal when the HTTP server closes all the connections. This allows to gracefully shutdown any application internal services, such as, for example, database connection pools:

const mysql = require('mysql');
const ws = require('x2node-ws');

const pool = mysql.createPool({
    connectionLimit: 5,
    host: process.env['DB_HOST'],
    port: process.env['DB_PORT'] || 3306,
    database: process.env['DB_NAME'],
    user: process.env['DB_USER'],
    password: process.env['DB_PASSWORD'],
    timezone: '+00:00'
});

ws.createApplication()
    .on('shutdown', () => {
        pool.end();
    })
    // configure the rest of the application
    // ...
    .run(Number(process.env['HTTP_PORT']));