pathdb

Database built on levelup/leveldb that stores javascript objects as a series of paths and values.

Installation

This module is installed via npm:

$ npm install pathdb

Background

In a typical key-value store such as leveldb, the entire 'object' is stored at a given key in the database. This is useful, but makes atomic updates to fields in an object more difficult, and prone to conflicts.

If the child fields in an object stored at a single key need to undergo a lot of updates from multiple clients, this creates a lot of read/lock/write operations.

In short, the finer grained the storage of fields is in the database, the lower the amount of contention and conflicts there will be.

PathDb was built from the ground up to support fine-grained storage of object properties, and to be able to replicate those changes in real-time to multiple clients, in the same way as say firebase, with real-time data bindings to client-side objects in angularjs.

Object paths vs Documents

Pathdb achieves this by taking an object such as:

// object to slice up
var o = {
  name: 'Eugene',
  number: 42,
  tags: ['tag1', 'tag2', 'tag3'],
  cars: [
    {
      make: 'Toyota',
      model: 'Camry'
    },
    {
      make: 'Toyota',
      model: 'Corolla'
    }
  ]
};

and slicing it into multiple key, value pairs, where the key represents the "path" to the value, and the value is the "leaf" value. So the object above becomes:

var paths =
  [ { key: [ 'name' ], value: 'Eugene' },
    { key: [ 'number' ], value: 42 },
    { key: [ 'tags', 0 ], value: 'tag1' },
    { key: [ 'tags', 1 ], value: 'tag2' },
    { key: [ 'tags', 2 ], value: 'tag3' },
    { key: [ 'cars', 0, 'make' ], value: 'Toyota' },
    { key: [ 'cars', 0, 'model' ], value: 'Camry' },
    { key: [ 'cars', 1, 'make' ], value: 'Toyota' },
    { key: [ 'cars', 1, 'model' ], value: 'Corolla' } ];

It does this using the pathos library.

Then, each of these individual slices are stored as individual key-value pairs in a levelup database.

Due to the magic of bytewise levelup custom encodings, all the slices that make up an object will sort next to each other in the leveldb database, making quick retrieval of objects and object trees using db.createReadStream.

The database as a single javascript object

By storing our objects using this path/value system, we can in a sense treat the entire database as as single javascript JSON object.

We can grab the entire object by fetching the whole database from the root:

db.pathdb.get([], function (err, obj) {
  // obj contains the whole database as an object
});

Or, more practically, we can fetch a subtree of the database:

db.pathdb.get(['my', 'path'], function (err, obj) {
  // obj contains the whole database as an object
});

Example Usage

Store and retrieve object

var pathdb = require('pathdb'),
    level = require('level'),
    bytewise = require('bytewise');

// db will be a levelup instance that has a 'pathdb' property with additional
// pathdb methods.

var db = pathdb(level('/my/db',
  { keyEncoding: bytewise, valueEncoding 'json'}));

// object to store
var person = {
  name: 'Eugene',
  number: 42,
  tags: ['tag1', 'tag2', 'tag3'],
  cars: [
    {
      make: 'Toyota',
      model: 'Camry'
    },
    {
      make: 'Toyota',
      model: 'Corolla'
    }
  ]
};

// store the object under the 'people' property
db.pathdb.put(['people'], o, cb);

// retrieve the stored object
db.pathdb.get(['people'], cb);

// fetch one of the child properties (will return 'Toyota')
db.get(['people', 'cars', 1, 'make'], cb);

// delete the object
db.pathdb.del(['people'], cb);

API

Requirements

Currently the pathdb database must be built on a typewise compliant levelup custom encoding such as bytewise.

pathdb(db)

Adds the pathdb property to an existing levelup instance.

var pathdb = require('pathdb'),
    bytewise = require('bytewise'),
    level = require('level');

// keyEncoding needs to be bytewise, valueEncoding needs to return a JS object
var db = level('/my/db', { keyEncoding: bytewise, valueEncoding: json });
// add pathdb functions to levelup
db = pathdb(db);

db.pathdb.put([path,] value, callback)

Will store the JSON object value at the location path in the pathdb object tree.

If the path is ommitted, then the object is stored at the root document level (ie. the entire database will be replaced by the contents of value).

db.pathdb.put(['my', 'path'], { name: 'Bob', number: 42 }, function (err) {
  // I/O or other error, pass it up the callback chain
  if (err) return callback(err);
});

db.pathdb.get([path, ], callback)

Will retrieve the JSON object subtree located at path. If the path is ommitted, then the entire database from the root will be retrieved.

db.pathdb.get(['my', 'path'], function (err, data) {
  if (err.name === 'NotFoundError') {
    // nothing found at the path
    return;
  }

  // I/O or other error, pass it up the callback chain
  if (err) return callback(err);
});

db.pathdb.del([path, ], callback)

Will delete the subtree located at path. If the path is ommitted, then the entire database will be deleted.

db.pathdb.del(['my', 'path'], function (err) {
  // I/O or other error, pass it up the callback chain
  if return callback(err);
});

db.pathdb.batch(path, array, callback)

Helper function to take a set of levelup batch commands with paths as keys, and store it at the appropriate path. This effectively prepends the path to the key attribute of each batch entry.

// this batch defines the creation of a new object:
//   { name: 'Eugene', number: 42 }

var batch = [
  [ { type: 'put', key: [ 'name' ], value: 'Eugene' },
    { type: 'put', key: [ 'number' ], value: 42 } ];

// store the object at the path [ 'my', 'path' ]
db.pathdb.batch(['my', 'path'], batch, function (err) {
  // I/O or other error, pass it up the callback chain
  if (err) return callback(err);
});

// The new object will effectively look like:
//   { my: { path: { name: 'Eugene', number: 42 } } }

db.pathdb.watch(path, default)

This function watches the object graph for subtree changes for anything at path or lower.

The function returns an EventEmitter which emits the following events:

• value - This gets emitted only once, and contains the initial value of the subtree at path. If there is nothing there, then the object defined by the second default parameter will be returned.
• change - This returns a changeset representing the changes made to the object defined by the path subtree.

This function is extremely useful for creating replication. Take a look at the replication unit tests for examples of using this method for replication in conjunction with the changeset and observejs modules.

Promise API

All the methods also return a Promise. Simply don't provide a callback parameter and the method will return a promise instead.

Eg:

var pathdb = require('pathdb'),
    levelPromise = require('level-promise'), // add promises to level db instance
    level = require('level'),
    bytewise = require('bytewise');

// db will be a levelup instance that has a 'pathdb' property with additional
// pathdb methods.

var db = pathdb(levelPromise(level('/my/db',
  { keyEncoding: bytewise, valueEncoding 'json'})));

// object to store
var person = {
  name: 'Eugene',
  number: 42,
  tags: ['tag1', 'tag2', 'tag3'],
  cars: [
    {
      make: 'Toyota',
      model: 'Camry'
    },
    {
      make: 'Toyota',
      model: 'Corolla'
    }
  ]
};

// store the object under the 'people' property
db.pathdb.put(['people'], o)
  .then(function () {
    console.log('success');
  })
  .catch(console.error);

// retrieve the stored object
db.pathdb.get(['people'])
  .then(function (data) {
    console.log(data);
  })
  .catch(console.error);

// fetch one of the child properties (will return 'Toyota')
db.get(['people', 'cars', 1, 'make'])
  .then(function (data) {
    console.log(data);
  })
  .catch(console.error);

// delete the object
db.pathdb.del(['people'])
  .then(function () {
    console.log('success');
  })
  .catch(console.error);

License

Copyright (c) 2016, Eugene Ware

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of Eugene Ware nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY EUGENE WARE ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL EUGENE WARE BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.