Shared Memory Objects

Super-fast file-based sharing of Javascript objects among multiple processes.

This module maps Javascript objects into shared memory for simultaneous access by different Node processes running on the same machine. Shared memory is loaded via mmap. Object access is mediated by Boost's unordered map class so object property access are speedy.

Data is lazily loaded piece-by-piece as needed so opening even a huge file takes no time at all.

There are two modes:

Unshared Write-only Mode

A single process creates a new file which is mapped to a Javascript object. Setting properties on this object writes those properties to the file. You can read from the object within this mode but sharing an object in write-only mode with other processes is certain to result in crashes.

Shared Read-only mode

Open an existing file for reading. Multiple processes can safely open this file. Opening is lightning fast and only a single copy remains in memory.

Faster performance with buffers

If you use lengthy data values, buffers can speed things up considerably. In rough benchmarking, a 300% speedup was see when reading 20k-byte values as buffers instead of strings. For 200k-byte values, the speedup was 2000%.

Requirements

Binaries are provided for OSX and Linux for various node versions (check the releases page to see which). If a binary is not provided for your platform, you will need Boost and and a C++11 compliant compiler (like GCC 4.8 or better) to build the module.

Installation

npm install mmap-object

Usage

const Shared = require('mmap-object')
const shared_object = new Shared.Create('filename')

shared_object['new_key'] = 'a string value'
shared_object.new_property = Buffer.from('a buffer value, supporting Unicode™')
shared_object['useless key'] = 0

// Erase a key
delete shared_object['useless_key']
shared_object.close()

// Read a file
const read_only_shared_object = new Shared.Open('filename')
console.log(`My value is ${read_only_shared_object.new_key}`)
console.log(`My other value is ${read_only_shared_object.new_property}`)

read_only_shared_object.close()

API

new Create(path, [file_size], [initial_bucket_count], [max_file_size])

Creates a new file mapped into shared memory. Returns an object that provides access to the shared memory. Throws an exception on error.

Arguments

• path - The path of the file to create
• file_size - Optional The initial size of the file in kilobytes. If more space is needed, the file will automatically be grown to a larger size. Minimum is 500 bytes. Defaults to 5 megabytes.
• initial_bucket_count - Optional The number of buckets to allocate initially. This is passed to the underlying Boost unordered_map. Defaults to 1024. Set this to the number of keys you expect to write.
• max_file_size - Optional The largest the file is allowed to grow in kilobites. If data is added beyond this limit, an exception is thrown. Defaults to 5 gigabytes.

Example

// Create a 500K map for 300 objects.
const obj = new Shared.Create('/tmp/sharedmem', 500, 300)

new Open(path)

Maps an existing file into shared memory. Returns an object that provides read-only access to the object contained in the file. Throws an exception on error. Any number of processes can open the same file but only a single copy will reside in memory. Uses mmap under the covers, so only those parts of the file that are actually accessed will be loaded.

Arguments

• path - The path of the file to open

Example

// Open up that shared file
const obj = new Shared.Open('/tmp/sharedmem')

close()

Unmaps a previously created or opened file. If the file was most recently opened with Create(), close() will first shrink the file to remove any unneeded space that may have been allocated.

It's important to close your unused shared files in long-running processes. Not doing so keeps shared memory from being freed.

The closing of very large objects (a few gigabytes and up) may take some time (hundreds to thousands of milliseconds). To prevent blocking the main thread, pass a callback to close(). The call to close() will return immediately while the callback will be called after the underlying munmap() operation completes. Any error will be given as the first argument to the callback.

Example

obj.close(function (err) {
  if (err) {
    console.error(`Error closing object: ${err}`)
  }
})

Iteration

The iterable protocol is supported for efficient iteration over the entire contents of the object:

const Shared = require('mmap-object')
const obj = new Shared.Open('filename')

for (let [key, value] of obj) {
    console.log(`${key} => ${value}`)
}

(This ES6 syntax is supported in Node 6+, for previous versions of node a more laborious syntax is necessary.)

isData()

When iterating, use isData() to tell if a particular key is real data or one of the underlying methods on the shared object:

const obj = new Shared.Open('/tmp/sharedmem')

for (let key in obj) {
  if (obj.isData(key)) { // Only show actual data
      console.log(key + ': ' + obj[key])
  }
}

isOpen()

Return true if this object is currently open.

isClosed()

Return true if this object has been closed.

get_free_memory()

Number of bytes of free storage left in the shared object file.

get_size()

The size of the storage in the shared object file, in bytes.

bucket_count()

The number of buckets currently allocated in the underlying hash structure.

max_bucket_count()

The maximum number of buckets that can be allocated in the underlying hash structure.

load_factor()

The average number of elements per bucket.

max_load_factor()

The current maximum load factor.

Unit tests

npm test

Limitations

It is strongly recommended to pass in the number of keys you expect to write when creating the object with Create. If you don't do this, the object will resize as you fill it up. This can be a very time-consuming process and can result in fragmentation within the shared memory object and a larger final file size.

Object values may be only string, buffer, or number values. Attempting to set a different type value results in an exception.

Symbols are not supported as properties.

Publishing a binary release

To make a new binary release:

• Edit package.json. Increment the version property.
• node-pre-gyp rebuild
• node-pre-gyp package
• node-pre-gyp-github publish
• npm publish

You will need a NODE_PRE_GYP_GITHUB_TOKEN with repo:status, repo_deployment and public_repo access to the target repo. You'll also need write access to the npm repo.

MSVS build prerequisites

Set up Boost.

Set BOOST_ROOT environment variable.

bootstrap
b2 --build-type=complete