datamodel-to-openapi specification

Data model to Open API Specification (FKA - Swagger) generator is an NPM module that generates OAS from a data model in JSON format.

Why automating generation of APIs?

Before getting into the how to use the framework, let's provide some context about it.

datamodel-to-openapi framework intends to solve DRY (don't repeat yourself) principle that many teams face when designing and building *data-oriented APIs. In my experience, every time API teams decide to expose a new resource, and it needs to debate over and over again the same design process. Through hundreds or even thousands of resources. You have a major bottleneck here, if you want to scale you API team(s), you need to find a better way automate this.

What if you could instead automate this process into something that could produce fast, more reliable and consistent results? Enter datamodel-to-openapi framework.

so, why the urgency of Data-Driven APIs?

The intent of Data-driven APIs is to address the following concerns:

• Consistency: APIs automatically generated by machines are more consistent than those produced by human beings by hand.
• Pluggability: As more data becomes available from underlying data sources, APIs expose these resources with minimal effort and in a controlled manner.
• Maintainability: your APIs evolve and require changes over time. Modifying hundreds of resources should not take more time than changing a single one.
• Affordance: By leveraging a data model as the foundation of your API, you provide access to your APIs from a data standpoint, similar to how we are used to accessing databases with SQL statements. Accessing your API resources should be similar. For instance, being able to filter, include or exclude attributes and nested entities, establish relationships and include hypermedia/HATEOAS automatically.
• Abstraction: APIs are becoming more mature by specification and standards produced by the industry. There is no need to invent a new standard. datamodel-to-openapi uses standard JSON as input to define a model along with its relationships. Then, it converts entities within the model into an OAS specification with paths, parameters, and annotation/vendor extensions with models. The OAS can be used as the input to integrate backend systems with abstraction layers. e.g. ORMs with Sequelize.js for relational databases, Mongoose for MongoDB for document oriented databases, SOAP stubs generated from WSDL files or other Web APIs via their SDKs.

so what is datamodel-to-openapi?

• It consists of a JSON file (data model JSON file, sample-data-model.json) with the definition of the data model that represents an API. For instance:
  • For an API management company, an API might consist of resources such as accounts, organizations, api proxies, etc.
  • For a retail company, it might look like a collection of resources such as products, orders, order items, vendors, etc.
  • For a banking company, products, services, customers, accounts, etc.
• what else contains a data model JSON file?
  • Entities, attributes, and relationships between them.
  • Also annotations or vendor extensions for:
    • Generating OAS paths, query params, headers, etc.
    • or used by other frameworks to automate building APIs even further
• A Node.js module and command-line tool that generates OAS from data model JSON file.

What does the output look like?

Check out test/swagger-sample-datamodel-to-oas-generated.json file

Getting Started

Installation

npm install datamodel-to-openapi -g

Using the CLI

Using the CLI is easy:

 $ git clone [email protected]:dzuluaga/datamodel-to-openapi.git
 $ cd datamodel-to-openapi
 $ datamodel-to-openapi generate ./test/sample-data-model.json

That's it. You should be able to pipe the output of the generated oas file to either a file or the clipboard with pbcopy.

Add a new resource

Let's add a new resource at the end of the JSON collection in sample-data-model.json and run again datamodel-to-openapi generate sample-data-model.json.

    {
      "model": "FoobarResource",
      "isPublic": true,
      "resources": {
        "collection": {
          "description": "A collection of Apigee APIs. TODO API Proxy definition.",
          "parameters": { "$ref": "./refs/parameters.json#/common/collection" },
          "responses": { "$ref": "./refs/responses.json#/apis/collection" }
        },
        "entity": {
          "description": "A single entity of an API Proxy.",
          "parameters": { "$ref": "./refs/parameters.json#/common/entity" },
          "responses": { "$ref": "./refs/responses.json#/apis/entity" }
        }
      },
      "name": "Foobar Resource",
      "path": "/foobarresource",
      "includeAlias": "FOOBARS",
      "listAttributes": {
        "id": { "type": "string", "is_primary_key": true, "alias": "foobar_resource_id" },
        "org_name": { "type": "string", "model": "Org", "description": "The org name."},
        "foobar_name": { "alias": "foobar_name", "type": "string", "is_secondary_key": true, "description": "The foobar name." },
        "created_date": {},
        "last_modified_date": {},
        "created_by": {},
        "last_modified_by": {}
      },
      "associations": [
        {
          "rel": "org",
          "foreignKey": "org_name",
          "modelName": "Org",
          "type": "belongsTo"
        }
      ],
      "schema": "schema_name",
      "table": "v_foobar"
    }

If you want to include /foobar resource as a subresource of /org/{org_id}, under Org model, include the following:

      "associations": [
        {
          "rel": "foobars",
          "foreignKey": "org_name",
          "modelName": "FoobarResource",
          "type": "hasMany"
        }
      ]

Voila! Your OAS Spec should now include resources GET /foobarresource, GET /foobarresource/{foobar_resource_id} and subresources GET /orgs/{org_name}/foobarresource and GET /orgs/{org_name}/foobarresource/{foobar_name}.

How can I visualize my OAS?

You can use Swagger-Editor and Swagger-UI. Here's how:

Try visualizing your resource by pasting the output in Swagger Editor.

You can also reference a remote Open API Spec by using the URL: http://editor.swagger.io/#/?import=https://raw.githubusercontent.com/dzuluaga/datamodel-to-openapi/master/test/swagger-sample-datamodel-to-oas-generated.json

Scroll all the way down to see the API. Ignore these errors in Swagger Editor code: "DUPLICATE_OPERATIONID" message: "Cannot have multiple operations with the same operationId: getResource". In our case, we do want multiple resources to leverage the same middleware function.

Another option to test your Open API Specification is by leveraging saving the output as a gist in Github and Swagger-UI to display it. Here's mine: http://petstore.swagger.io/?url=https://gist.githubusercontent.com/dzuluaga/b5e87ba7829f3305ad8d5bbbd7d71255/raw/f1a2cd51c09d0ca6eb6ed65d99900b2bf6273737/oas-datamodel-to-oas-generated.json

Using the API

The following example can be found under test/app.js:

var datamodelToSwagger = require('../index');

datamodelToOas.generateOas( require('./sample-data-model.json') )
    .then( function( oasDoc ) {
      console.log( JSON.stringify( oasDoc, null, 2 ) );
    })
    .catch( function( err ) {
      console.log( err.stack );
    });

The Node.js module returns a promise with an Open API Specification resolving sample-data-model.json.

How can I take this even further with an API and a database

As mentioned earlier, the OAS generated by this tool includes x-data-model annotations for each path. Another layer leverages these annotations that the API uses to lookup models to support http verbs. This adapter layer can represent anything that it can interact with. So, not necessarily a database. For instance other services such as Web Services, other REST APIs, FTP servers, you name it.

The implementation of this adapter layer is up to you. However, I built Nucleus-Model-Factory, a lightweight framework on top of Sequelize.js for the purpose of supporting Postgres database.

The Database/Backend Adapter

• For an example of a Sequelize.js adapter check this example.

Node.js API Example

An example of a Node.js app leveraging the data model to generate the OAS spec and Postgres models (Sequelize.js ORM):

'use strict';

var app = require('express')(),
    path = require('path'),
    all_config = require('./config.json'),
    utils = require('nucleus-utils')( { config: all_config }),
    config = utils.getConfig(),
    modelFactory = require('nucleus-model-factory'),
    dataModelPath = './api/models/edge-data-model.json',
    edgeModelSpecs = require( dataModelPath),
    http = require('http'),
    swaggerTools = require('swagger-tools'),
    dataModel2Oas = require('datamodel-to-openapi'),

var serverPort = 3000;

// swaggerRouter configuration
var options = {
  controllers: './api/routers',
  useStubs: process.env.NODE_ENV === 'development' ? true : false // Conditionally turn on stubs (mock mode)
};

dataModel2Oas.generateOasAt( dataModelPath )
    .then( function( oasDoc ) {
      swaggerTools.initializeMiddleware( oasDoc , function (middleware) {
        var models = modelFactory.generateModelMap( edgeModelSpecs, utils );
        if( !models ){ throw new Error('No models were found. Check models.json') }
        utils.models = models;

        if( !oasDoc['x-db-models-var-name'] ) { throw new Error('Undefined x-db-model-var-name attribute in swagger spec at root level'); }
        app.set( oasDoc['x-db-models-var-name'], models );

        // Interpret Swagger resources and attach metadata to request - must be first in swagger-tools middleware chain
        app.use(middleware.swaggerMetadata());

        // Validate Swagger requests
        app.use(middleware.swaggerValidator());

        // Route validated requests to appropriate controller
        app.use( middleware.swaggerRouter(options) );

        // catch 404 and forward to error handler
        app.use(function(req, res, next) {
          var err = new Error('Not Found');
          err.status = 404;
          next(err);
        });

      });
    })
    .then( function() {
      // Start the server
      http.createServer(app).listen(serverPort, function () {
        console.log('Your server is listening on port %d (http://localhost:%d)', serverPort, serverPort);
      });
    })
    .catch( function( err ) {
      console.log( err.stacktrace );
    })

The Router/Controller

The router/controller is referenced by each path generated within the OAS by datamodel-to-openapi. Then based on x-data-model annotation, it will know how to execute query based on all parameters and whereAttributes annotation.

module.exports.getResource = function getResource (req, res, next) {
  var datamodel = req.swagger.operation[ 'x-data-model' ];
  debug( datamodel );
  middlewareBuilder( req, res, next, datamodel );
};

function middlewareBuilder( req, res, next, datamodel ) {
  var utils = req.app.get('utils');
  debug('use spec', datamodel);
  var where = mergeAndExtractParams(datamodel, req);
  debug('WHERE', JSON.stringify(where));
  debug('use model', datamodel);
  var model = req.app.get('edge_models')[datamodel.model];
  if (!model) {
    next("Model name not found : " + datamodel.model);
  }
  else {
    if (req.query.describe && req.query.describe === 'true') { // returns table description
      res.json({"message": "not implemented yet"});
      /*options.model.describe()
       .then( function( describe ){
       res.json( describe );
       } )*/
    } else {
      model[datamodel.cardinality]({
        where: where,
        attributes: utils.tryToParseJSON(req.query.attributes, utils.messages.PARSE_ERROR_ATTRIBUTE_PARAM, model.listAttributes),
        offset: req.query.offset || 0,
        limit: utils.getLimit(req.query.limit),
        order: req.query.order || [],
        include: utils.getIncludes(utils.models, req.query.include)
      })
          .then(function (items) {
            if (!items || items.length == 0) res.status(404).json({code: "404", message: "Resource not found."});
            else res.json({entities: items});
          })
          .catch(function (error) {
            utils.sendError("500", error, req, res);
          });
    }
  }
};

/*
 * Dynamically generates where object with attributes from the request object
 */
function mergeAndExtractParams(routeSpec, req ){
  var _where = { };
  var utils = req.app.get('utils');
  debug('mergeAndExtractParams', routeSpec.whereAttributes);
  ( routeSpec.whereAttributes || [] ).forEach( function( attr ) {
    var operator = attr.operator || "$eq";
    _where[ attr.attributeName ] = { };
    var value = req.swagger.params[ attr.paramName].value;
    // if operator is like concatenate % before and after
    if( operator === '$like' ){
      value = '%'.concat( value.concat('%'));
    }
    _where[ attr.attributeName ][operator] = value;//req.params[ attr.paramName ];
  } );
  debug("req.query", req.query);
  var where = utils.db_connections.sequelize.Utils._.merge( _where, utils.tryToParseJSON( req.query.where, utils.messages.PARSE_ERROR_WHERE_PARAM, null ) );
  debug('extractParams_before_merged', where);
  //where = utils.db_connections.sequelize.Utils._.merge( where, applySecurity(  routeSpec, req.security.account_list, where ) );
  debug('extractParams_merged', where);
  return where;
}

function applySecurity( options, account_list, where ) {
  debug('applySecurity', options.securityAttributeName);
  debug('applySecurity', account_list);
  var _where = {}
  if( account_list && account_list.length > 0 && account_list[0] !== '*' ){
    _where[ options.securityAttributeName || 'account_id' ] = { $in: account_list };
  } else if( !account_list ){
    throw new Error("User Credentials require user account mapping.");
  }
  return _where;
}

Are there any examples of APIs using this paradigm?

Yes. This framework is the product of drinking-our-kool-aid in my team. So, we're continuously adding new features and improving it.

Summary

By following above steps, we've just achieved automation of building of our API based while adhering to the principles of data-driven APIs. Trying to do this by hand would be untenable.

References

• The sample code introduces a few steps that aren't necessarily documented in this readme file. It is recommended to check out their documentation. For (Swagger-Tools)[https://github.com/apigee-127/swagger-tools] and (Nucleus-Model-Factory)[https://www.npmjs.com/package/nucleus-model-factory] take either the datamodel.json file or the output of datamodel-to-openapi.

• *This framework is based on (data-oriented API principles)[http://apigee.com/about/blog/developer/data-oriented-designs-common-api-problems] introduced by Martin Nally and Marsh Gardiner. Also check this (webcast)[http://apigee.com/about/resources/webcasts/pragmatic-rest-next-generation].

TODO

• [ ] datamodel-to-openapi currently lacks the support of verbs that modify resource state. Although, it could be extended to meet those needs. Stay tuned for this functionality.

Feedback and pull requests

Feel free to open an issue to submit feedback or a pull request to incorporate features into the main branch.