node-virtdb-provider (v2.3)

Overview

This is a node.js module that helps creating new data connectors to the VirtDB system.

How to write a new data connector

By using the virtdb-provider module it is quite easy to create new connectors. Only the METADATA and the DATA interfaces have to be implemented. To get detailed information of the structures used in the following examples see the starschema/virtdb-proto project.

METADATA

The METADATA interface is used for letting VirtDB know the structure of the data provided. This is done by replying to METADATA requests.

# A typical meta_data request.
request =
    Name: "customers"
    WithFields: false
}

# A typical meta_data reply
reply =
    Tables: [
        Name: "customers"
        Fields: [
            Name: "id"
            Desc:
                Type: 'UINT64'
        ,
            Name: "name"
            Desc:
                Type: 'STRING'
        ]
    ]

DATA

The DATA interface is used for sending data back to queries. Data is sent in column chunks.

# A typical query
query =
    QueryId: "e142fbd0-9d77-11e4-bd06-0800200c9a66"
    Table: "customers"
    Fields: [
        Name: "id"
        Desc:
            Type: 'UINT64'
    ,
        Name: "name"
        Desc:
            Type: 'STRING'
    ]
    Filter: [
        Operand: "like"
        Simple:
            Variable: "name"
            Value: "John%"
    ]
}

# A typical column reply for the query. (Separate replies are needed for each columns requested in the query.)
columnChunk =
    QueryId: "e142fbd0-9d77-11e4-bd06-0800200c9a66"
    Name: "name"
    SeqNo: 0 # a strictly increasing series for each column
    Data:
        Type: 'STRING'
        StringValue: ['John Doe', 'John Smith', 'Johnathan Apple']
    EndOfData: true
    CompType: "NO_COMPRESSION" # alternative: LZ4_COMPRESSION

Sample implementation

DataProvider = require 'virtdb-provider'

# component name should be unique throughout the system
# url is the ZeroMQ url of the virtdb config service component.
virtdb = new DataProvider('<component-name>', '<url>')

# meta_data is a list of table descriptions that match the request criteria
virtdb.onMetaDataRequest (request, sendFunction) ->
    metadata = getMetadataFromYourSource request
    sendFunction null, metadata
    return

# virtdb data is sent by column chunks
virtdb.onQuery (request, callback) ->
    data = queryYourSource request
    reply = virtdb.getDataHandler request
    for row in data
        reply.pushObject row
    reply.send()
        
# template contains the structure of the configuration you demand
template = 
    AppName: '<component-name>'
    Config: [
        VariableName: '<key>'
        Type: 'STRING'
        Scope: '<scope>'
    ]

# with this method you register your config template in the system and set the function where you will receive the configuration on    
virtdb.registerConfig template, (config) ->
    console.log "Yay, new config: #{config}"

VirtDBDataProvider

constructor (name, connectionString, callback)

Connects to the VirtDB system.   

• name: the name of component. This name has to be unique throughout a VirtDB installation.
• connectionString: the ZeroMQ connection string of the Endpoint service socket of the Config Service component.
• callback: function callback(err) A function that is called when connection is finished. err is defined if something went wrong.

onQuery(callback)

Calls the given callback when a query request is received.

• callback: function callback(request)

onMetaDataRequest(requestHandler, options, callback)

Calls the given callback when a meta data request is received.

• requestHandler(envelope, request, sendMethod): The callback that will be called when Metadata request arrives.
  • envelope: The id of the message. It should be passed to the sendMethod on sending the reply.
  • request: The metadata request describing the requested table and if the reply should contain the fields beside the table names.
  • sendMethod(err, envelope, metadata): The method that needs to be called when the metadata is available.
• options: The options for the server that is to be initialized. For the possible options and their meanings see the virtdb-connector project.
• callback(err, address): A callback that is called when an error happens during the setup or the server is set up and listening.

registerConfig(template, callback)

Makes the component configurable through the VirtDB config protocol. For a detailed protocol description see the [virtdb-proto](https://github.com/starschema/virtdb-proto) project.

• template: The configuration options of the component.
• callback: function callback(config) Called when configuration messages arrive.

createTable(tableName, schema)

Creates a new VirtDBTable object which helps to send a response to a MetaData request.

• name: the name of the table.
• schema: the name of the scheam which tha table belongs to.

getDataHandler(request)

Creates a new VirtDBReply object which helps to send a response to a Query request. Returns null if no table with the name and schema in the Query has been created before with createTable. The created table also have to contain the fields that are requested in the Query.

• query: the incoming Query request.

close()

Closes sockets and connections.

detectFieldType(sample)

Static. Detects the field type of the column through a data sample.

• sample: an array of data from the column which we want to detect.

VirtDBData

constructor (query)

Creates a new VirtDBReply object which helps to send a response to a Query request.

• query: the incoming Query request.

pushObject(row)

Adds a data row to the reply.

• row: Key-value pairs where the key is the identifier of the column and the value is the value in the row.

send(endOfData, seqNo)

Sends the Column response.

• endOfData: Default: true. If it is false it indicates the response is only a part of the whole response.
• seqNo: optional: In some cases it is more efficient to send the data as it arrives even if it is not ordered. In this case seqNo can't be figured out by VirtDBData and has to be set externally. First seqNo must always be 0 and seqNo is strictly increasing.

#VirtDBTable ##

constructor (name, schema)

Creates a new VirtDBTable object which helps to send a response to a MetaData request.

• name: the name of the table.
• schema: the name of the scheam which tha table belongs to.

addField(name, type)

Adds a field to the table.

• field: the name of the field or the data column.
• type: the type of the field. For possible values see the common.proto file in the starschema/virtdb-proto project.

Credits

VirtDB is the product of Starschema Technologies.. For any questions, comments or issues please contact us: @virtdb