Welcome to the R5N DHT Repository

Project Lead: Dave Hagman aka avatar_dave

Project Start: 2/8/2014

Project Description: A JavaScript DHT implementation using the R5N routing algorithm.

Current development notes: Since this is in active development this code should not be used in any production applications. The code is still in Alpha stage.

Introduction

R5N is a Javascript implementation of a Distributed Hash Table using the R5N routing algorithm. This module has all the functionality of a DHT plus the ability to define custom message handlers.

Configuring the DHT

All configuration options are stored in a Globals object (/lib/globals.js). Some of the common, safe configuration values are below. We call them safe if it is something the developer can change without serious implications to the DHT. Any configuration setting not mentioned below should not be changed unless you are 100% sure of the implications.

Category: Network

• DHT_PUBLIC_ADDRESS - The public address for the DHT (required)
• DHT_INIT_PORT - The start port for the DHT. This is the port that the DHT will bind a listener socket to. (required)
• ESTIMATED_PEER_COUNT - The estimated number of peers on the network. This should be adjusted if the network grows or shrinks by any significant amount.
• ESTIMATED_CONN_PER_PEER - The estimated amount of connections a peer will have at any one time.

Category: KeyProps

• DEFAULT_KEY_LENGTH - The default length for generated keys such as client ID and node ID.

Category: RecursiveRouting

• MAX_FIND_VALUE_ATTEMPTS - The maximum number of times to retry a failed FIND_VALUE operation.

Category: AsyncTasks

• CLEAR_PING_REQ_QUEUE - Interval for clearing the failed ping request queue and clearing stale nodes.
• CLEAR_PING_REQ_QUEUE - Interval (in ms) for performing replication of data to peer nodes.

Starting the DHT

The process for starting the DHT differs upon whether we are "bootstrapping" off a running DHT node or we are spinning up a new DHT.

Creating the first node

In this scenario we are creating a new DHT. All we need to do is instantiate a new DHT and start it. Once other nodes are spun up we need to follow the bootstrap process below.

var DHT = require(__dirname + '/lib/dht');

// Put the IP address and port for the DHT here
// NOTE: This is the public IP for this node, not localhost or 127.0.0.1
var address = 'IP_ADDRESS';
var port = 3000;

// Instantiate new DHT and start it
var dht = new DHT({
    IP: address,
    port: port
});
dht.start().then(function(res) {
	console.log('DHT started.');
});

Bootstrapping off an existing DHT node

In this scenario, we have an existing DHT and we just want to add a new node. In order to do that we need to "bootstrap" it off of another node using the boostrap() method. The bootstrap() method returns a promise object.

var DHT = require(__dirname + '/lib/dht');

// Put the IP address and port for the DHT here
// NOTE: This is the public IP for this node, not localhost or 127.0.0.1
var address = 'IP_ADDRESS';
var port = 3000;

// Instantiate new DHT and start it
var dht = new DHT({
    IP: address,
    port: port
});
dht.start().then(function(res) {
	console.log('DHT started.');
});

// Bootstrap off existing DHT. Internally, this sends a FIND_NODE message to 
// the other DHT which then returns a list of closest peers from it's routing table
// and also gets added to the routing tables of it's closest peers
// NOTE: You have to know the IP address and port of an existing DHT
dht.boostrap('OTHER_NODE_ADDRESS', 3000)
    .then(function(res) {
        console.log('Bootstrap complete with %d new routes.', res.number_of_routes);
    })
    .catch(function(err) {
        console.log('ERROR: %s', err);
    })
    .done();

Storing Data

Data in a DHT is stored according to an XOR Distance metric. Each Node ID is compared to the key and whichever Node is closest to the key stores the data.

To store data, pass a key/value pair to the store() method on the DHT. A promise is returned where you can perform post processing and error handling for the RPC.

dht.store(key, value)
    .catch(function(err) {
        console.log('Error in STORE RPC: %s', err);
    })
    .then(function(res) {
        console.log('Store complete!\nKey %s was stored at node ID: %s', res.rpc.get('key'), res.rpc.get('stored_at'));
    });

As long as there were no errors, the log above would print:

Store complete! Key {SOME_KEY_FOR_VALUE} was stored at node ID: {NODE_ID}

Retrieving Data

Data in a DHT is retrieved the same way it is stored. To retrieve data, pass a key to the findValue() method on the DHT. A deferred promise is returned. If the key is found, the promise is resolved and the success function is called. If not the promise is rejected and the error function is called.

dht.findValue(key)
    .then(
        function(res) {
            // Success!
            console.log('Find complete!\nValue %s was retrieved at node ID: %s', res.rpc.get('value'), res.rpc.get('found_at'));
        },
        function(err) {
            // No value found for that key
            console.log('No values found.');
        }
    );

As long as a value was found, the log below would look like this:

Find complete! Value {SOME_VALUE} was retrieved at node ID: {NODE_ID}

Adding custom message handlers

The need may arise when you need to add custom functionality to the DHT such as a new message type. This R5N implementation has an API for adding custom messages and handlers without having to modify the core DHT code. The custom message API employs a decorator pattern to accomplish this.

The DHT has a method called addCustomMessageHandler(). This method takes an object as a parameter with the following properties: messageType, onMessage

• The messageType property is a string for the message type (such as 'get_random_kvs'). This is used throughout the lifecycle of an RPC to identify the message type.
• The onMessage property is a callback function. This callback is called when the DHT receives an RPC with the above message type. This is where you handle the message and you can choose to forward it to another DHT or end it and send the response back to the initiating node.

Once you have configured your custom message type, you can initiate a request of that type with the sendCustomRPC() method on the DHT. This method takes 2 parameters:

• The message type string (like 'get_random_kvs')
• An RPC object

The sendCustomRPC() method returns a promise and you can process the response when it gets resolved (see the example below). Combining the custom message API with the flexible data container in the RPC makes this solution extremely flexible. We have already implemented custom message types for Avatar using only this API. Custom message types for the DHT are modular and new additions are highly unlikely to break existing functionality.

For the sake of the example below, assume we have a DHT instance running. The DHT instance is associated with the variable named 'dht' (for obvious reasons). This is how you would configure a DHT to handle custom message types.

// This is our setup object
// We need to initialize the 3 properties stated above
 var customMessageOptions = {
	// This is the unique message type string
	// It can be whatever you want it to be
	// NOTE: This is important because this is how we identify and handle RPC's of this type.
	// Consider keeping this in a global constant somewhere in your app
	messageType: 'my_custom_message_type',
	onMessage: function(rpc) {
		
		// Here we can process a message for 'my_custom_message_type'
		// The onMessage and onResponse methods are called within the context
		// of the DHT so using the 'this' reference is actually pointing to the
		// DHT currently calling your callback. That makes it easy to use any of
		// the DHT functionality. For example, if we wanted to access the DHT's routing
		// table to get a list of all routes, we could do this:
		var allRoutes = this.routingTable.getAllRoutes();
		
		// Do something with the routes here
		
	}
};

// Add the custom message to the DHT
dht.addCustomMessageHandler(customMessageOptions);

The DHT is now configured to handle message types of my_custom_message_type. But what if you want to initiate a request with that type? See below.

	// To send a custom message, use the sendCustomRPC method
	// NOTE: This will throw an exception if you haven't setup handlers for this message type
	var rpc = new RPC();
    rpc.address = 'SOME_IP_ADDRESS';
    rpc.port = 9999;
	
	// Let's add some arbitrary data to the RPC specific to this message type
	rpc.set('some_random_data', 123456);
	
	// Initiate the request
    dht.sendCustomRPC('my_custom_message_type', rpc)
    .then(function(result) {
        // Here we process the response. In the custom message API, the onResponse
        // method is only called on the initiating node. This mimicks the request/response
        // feel of typical HTTP functionality. 
        console.log('RECEIVED CUSTOM RESPONSE: %j', result.rpc);
    });