LMDB for Node.js

A Low-level, LevelDOWN-compatible, Node.js LMDB binding

"LMDB" is Symas Lightning Memory-Mapped Database.

LMDB is an ultra-fast, ultra-compact key-value embedded data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases, and is only limited to the size of the virtual address space, (it is not limited to the size of physical RAM). Note: LMDB was originally called MDB, but was renamed to avoid confusion with other software associated with the name MDB

LMDB for Node.js, is primarily designed to serve as a back-end to LevelUP, it is strongly recommended that you use LevelUP in preference to this library directly.

Tested & supported platforms

• Linux
• Others... testing soon

API

• lmdb()
• lmdb#open()
• lmdb#close()
• lmdb#put()
• lmdb#get()
• lmdb#del()
• lmdb#batch()
• lmdb#approximateSize()
• lmdb#getProperty()
• lmdb#iterator()
• iterator#next()
• iterator#end()
• lmdb.destroy()
• lmdb.repair()

lmdb(location)

lmdb() returns a new Node.js LMDB instance. location is a String pointing to the LMDB store to be opened or created.

lmdb#open([options, ]callback)

open() is an instance method on an existing database object.

The callback function will be called with no arguments when the database has been successfully opened, or with a single error argument if the open operation failed for any reason.

options

The optional options argument may contain:

• 'createIfMissing' (boolean, default: true): If true, will initialise an empty data store at the specified location if one doesn't already exist. If false and a database doesn't exist you will receive an error in your open() callback and your database won't open.

• 'errorIfExists' (boolean, default: false): If true, you will receive an error in your open() callback if the database exists at the specified location.

• 'mapSize' (boolean, default: 10485760 (10MB)): Specify the size of the memory map, which is also the maximum size of the database. The value should be chosen as large as possible, to accommodate future growth of the database. The size may be changed by closing and reopening the environment. Any attempt to set a size smaller than the space already consumed by the environment will be silently changed to the current size of the used space. The size should be a multiple of the OS page size.

• 'maxReaders' (integer, default: 126): Specify the number of slots in the readers table, which determines the maximum number of concurrent iterators. The reader table is stored in the lock file and occupies 64 bytes per slot. Iterators error with MDB_READERS_FULL when there are no free reader slots, so set this to allow space for as many concurrent iterators as you expect to use. The size may be changed by closing and reopening the environment.

• 'sync' (boolean, default: true): By default, system buffers are flushed to disk after committing transactions (which are performed on every operation). Use this option to turn off this behaviour to speed up writes at the risk of losing writes upon system crash. Note that setting 'sync' to false and 'writeMap' to true leaves the system with no hint for when to write transactions to disk. Using 'mapAsync' with 'writeMap' may be preferable.

• 'readOnly' (boolean, default: false): Open the environment in read-only mode. No write operations will be allowed. LMDB will still modify the lock file - except on read-only filesystems, where LMDB does not use locks.

• 'writeMap' (boolean, default: false): Use a writeable memory map (unless 'readOnly' is set). This is faster and uses fewer malloc operations. Note that setting 'writeMap' to true will involve the pre-allocation of the data store, as a single file, of 'mapSize' size.

• 'metaSync' (boolean, default: true): System buffers will be flushed to disk once per transaction (unless 'sync' is set to false). Setting 'metaSync' to false will prevent a metadata flush, deferring it until the system flushes files to disk, or the next (non-read-only) write. A 'metaSync' set to false will improve performance and maintain database integrity, but a system crash may undo the last committed transaction.

• 'mapAsync' (boolean, default: false): When using a 'writeMap', use asynchronous flushes to disk. As with 'sync' set to false, a system crash can then corrupt the database or lose the last transactions.

lmdb#close(callback)

close() is an instance method on an existing database object. The underlying LMDB database will be closed and the callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.

lmdb#put(key, value[, options], callback)

put() is an instance method on an existing database object, used to store new entries, or overwrite existing entries in the LMDB store.

The key and value objects may either be Strings or Node.js Buffer objects and cannot be undefined or null. Other object types are converted to JavaScript Strings with the toString() method and the resulting String may not be a zero-length. A richer set of data-types are catered for in LevelUP.

The callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.

lmdb#get(key[, options], callback)

get() is an instance method on an existing database object, used to fetch individual entries from the LMDB store.

The key object may either be a String or a Node.js Buffer object and cannot be undefined or null. Other object types are converted to JavaScript Strings with the toString() method and the resulting String may not be a zero-length. A richer set of data-types are catered for in LevelUP.

options

• 'asBuffer' (boolean, default: true): Used to determine whether to return the value of the entry as a String or a Node.js Buffer object. Note that converting from a Buffer to a String incurs a cost so if you need a String (and the value can legitimately become a UFT8 string) then you should fetch it as one with asBuffer: true and you'll avoid this conversion cost.

The callback function will be called with a single error if the operation failed for any reason. If successful the first argument will be null and the second argument will be the value as a String or Buffer depending on the asBuffer option.

lmdb#del(key[, options], callback)

del() is an instance method on an existing database object, used to delete entries from the LMDB store.

The key object may either be a String or a Node.js Buffer object and cannot be undefined or null. Other object types are converted to JavaScript Strings with the toString() method and the resulting String may not be a zero-length. A richer set of data-types are catered for in LevelUP.

The callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.

lmdb#batch(operations[, options], callback)

batch() is an instance method on an existing database object. Used for very fast bulk-write operations (both put and delete). The operations argument should be an Array containing a list of operations to be executed sequentially, although as a whole they are executed within a single transaction on LMDB. Each operation is contained in an object having the following properties: type, key, value, where the type is either 'put' or 'del'. In the case of 'del' the 'value' property is ignored. Any entries with a 'key' of null or undefined will cause an error to be returned on the callback and any 'type': 'put' entry with a 'value' of null or undefined will return an error. See LevelUP for full documentation on how this works in practice.

The callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.

lmdb#iterator([options])

iterator() is an instance method on an existing database object. It returns a new Iterator instance which abstracts an LMDB "cursor".

options

The optional options object may contain:

• 'start': the key you wish to start the read at. By default it will start at the beginning of the store. Note that the start doesn't have to be an actual key that exists, LMDB will simply jump to the next key, greater than the key you provide.

• 'end': the key you wish to end the read on. By default it will continue until the end of the store. Again, the end doesn't have to be an actual key as an (inclusive) <=-type operation is performed to detect the end. You can also use the destroy() method instead of supplying an 'end' parameter to achieve the same effect.

• 'reverse' (boolean, default: false): a boolean, set to true if you want the stream to go in reverse order.

• 'keys' (boolean, default: true): whether the callback to the next() method should receive a non-null key. There is a small efficiency gain if you ultimately don't care what the keys are as they don't need to be converted and copied into JavaScript.

• 'values' (boolean, default: true): whether the callback to the next() method should receive a non-null value. There is a small efficiency gain if you ultimately don't care what the values are as they don't need to be converted and copied into JavaScript.

• 'limit' (number, default: -1): limit the number of results collected by this iterator. This number represents a maximum number of results and may not be reached if you get to the end of the store or your 'end' value first. A value of -1 means there is no limit.

• 'keyAsBuffer' (boolean, default: true): Used to determine whether to return the key of each entry as a String or a Node.js Buffer object. Note that converting from a Buffer to a String incurs a cost so if you need a String (and the value can legitimately become a UFT8 string) then you should fetch it as one.

• 'valueAsBuffer' (boolean, default: true): Used to determine whether to return the value of each entry as a String or a Node.js Buffer object.

iterator#next(callback)

next() is an instance method on an existing iterator object, used to increment the underlying LMDB cursor and return the entry at that location.

the callback function will be called with no arguments in any of the following situations:

• the iterator comes to the end of the store
• the end key has been reached; or
• the limit has been reached

Otherwise, the callback function will be called with the following 3 arguments:

• error - any error that occurs while incrementing the iterator.
• key - either a String or a Node.js Buffer object depending on the keyAsBuffer argument when the createIterator() was called.
• value - either a String or a Node.js Buffer object depending on the valueAsBuffer argument when the createIterator() was called.

iterator#end(callback)

end() is an instance method on an existing iterator object. The underlying LMDB cursor will be deleted and the callback function will be called with no arguments if the operation is successful or with a single error argument if the operation failed for any reason.

Getting support

There are multiple ways you can find help in using LMDB / LevelUP / LevelDB in Node.js:

• IRC: you'll find an active group of LevelUP users in the ##leveldb channel on Freenode, including most of the contributors to this project.
• Mailing list: there is an active Node.js LevelUP Google Group.
• GitHub: you're welcome to open an issue here on this GitHub repository if you have a question.

Contributing

Node.js LMDB is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the CONTRIBUTING.md file for more details.

Licence & copyright

Copyright (c) 2012-2013 Node.js LMDB contributors.

Node.js LMDB is licensed under an MIT +no-false-attribs license. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE file for more details.

Node.js LMDB builds on the excellent work of the Howard Chu of Symas Corp and additional contributors. LMDB are both issued under the The OpenLDAP Public License.