many-level
v2.0.0
Published
Share an abstract-level database over the network or other kind of stream
Downloads
44,102
Readme
many-level
Share an abstract-level database over the network or other kind of stream. The successor to multileveldown
. If you are upgrading, please see UPGRADING.md.
:pushpin: Which module should I use? What is
abstract-level
? Head over to the FAQ.
Usage
Use this module to share an abstract-level
database across multiple processes or machines. A host exposes a database of choice over binary streams, using compact Protocol Buffers messages to encode database operations. One or more guests connect to that host and expose it as an abstract-level
database, as if it is a regular, local database. They can opt-in to a seamless retry in order to reconnect to a host without aborting any pending database operations.
First create a host and server. The server can be anything that supports binary streams. In this example we'll use a simple TCP server.
const { ManyLevelHost } = require('many-level')
const { Level } = require('level')
const { pipeline } = require('readable-stream')
const { createServer } = require('net')
const db = new Level('./db')
const host = new ManyLevelHost(db)
const server = createServer(function (socket) {
// Pipe socket into host stream and vice versa
pipeline(socket, host.createRpcStream(), socket, () => {
// Disconnected
})
})
server.listen(9000)
Then create some guests:
const { ManyLevelGuest } = require('many-level')
const { pipeline } = require('readable-stream')
const { connect } = require('net')
const db = new ManyLevelGuest()
const socket = connect(9000)
// Pipe socket into guest stream and vice versa
pipeline(socket, db.createRpcStream(), socket, () => {
// Disconnected
})
await db.put('hello', 'world')
console.log(await db.get('hello'))
Encoding options are supported as usual.
Reconnect
To setup reconnect set the retry
option to true
and reconnect to your server when the connection fails:
const db = new ManyLevelGuest({
retry: true
})
const reconnect = function () {
const socket = connect(9000)
pipeline(socket, db.createRpcStream(), socket, () => {
// Reconnect after 1 second
setTimeout(reconnect, 1000)
})
}
reconnect()
The guest database will now retry your pending operations when you reconnect. If you create an iterator or readable stream and your connection fails halfway through reading that iterator then many-level
makes sure to only retry the part of the iterator that you are missing. The downside of retry
is that a guest database then cannot provide snapshot guarantees because new iterators (and thus snapshots of the host database) will be created upon reconnect. This is reflected in db.supports.snapshots
which will be false
if retry
is true.
API
Host
host = new ManyLevelHost(db, [options])
Create a new host that exposes the given db
, which must be an abstract-level
database that supports the 'buffer'
encoding (most if not all do). It can also be a sublevel which allows for exposing only a specific section of the database.
The optional options
object may contain:
readonly
(boolean, defaultfalse
): reject write operations likedb.put()
preput
(function, default none): afunction (key, val, cb) {}
to be called beforedb.put()
operationspredel
(function, default none): afunction (key, cb) {}
to be called beforedb.del()
operationsprebatch
(function, default none): afunction (operations, cb) {}
to be called beforedb.batch()
operations.
hostStream = host.createRpcStream()
Create a duplex host stream to be piped into a guest stream. One per guest.
Guest
db = new ManyLevelGuest([options])
Create a guest database that reads and writes to the host's database. The ManyLevelGuest
class extends AbstractLevel
and thus follows the public API of abstract-level
with a few additional methods and one additional constructor option. As such, the majority of the API is documented in abstract-level
. It supports sublevels, iterator.seek()
and every other abstract-level
feature, except for the createIfMissing
and errorIfExists
options which have no effect here. Iterators have built-in end-to-end backpressure regardless of the transport that you use.
The optional options
object may contain:
keyEncoding
(string or object, default'utf8'
): encoding to use for keysvalueEncoding
(string or object, default'utf8'
): encoding to use for valuesretry
(boolean, defaultfalse
): if true, resend operations when reconnected. If false, abort operations when disconnected, which means to yield an error on e.g.db.get()
.
The database opens itself but (unlike other abstract-level
implementations) cannot be re-opened once db.close()
has been called. Calling db.open()
would then yield a LEVEL_NOT_SUPPORTED
error.
guestStream = db.createRpcStream([options])
Create a duplex guest stream to be piped into a host stream. Until that's done, operations made on db
are queued up in memory. Will throw if createRpcStream()
was previously called and that stream has not (yet) closed. The optional options
object may contain:
ref
(object, defaultnull
): an object to only keep the Node.js event loop alive while there are pending database operations. Should have aref()
method to be called on a new database operation likedb.get()
and anunref()
method to be called when all operations have finished (or when the database is closed). A Node.jsnet
socket satisfies that interface. Theref
option is not relevant whenManyLevelGuest
is used in a browser environment (which, side note, is not officially supported yet).
guestStream = db.connect()
An alias to createRpcStream()
for multileveldown
API compatibility.
db.forward(db2)
Instead of talking to a host, forward all database operations to db2
which must be an abstract-level
database. This method is used by rave-level
and serves a narrow use case. Which is to say, it may not work for anything other than rave-level
. Among other things, it assumes that db2
is open and that it uses the same encoding options as the guest database.
db.isFlushed()
Returns true
if there are no operations pending to be sent to a host.
Install
With npm do:
npm i many-level
Usage from TypeScript also requires npm install @types/readable-stream
.
Contributing
Level/many-level
is an OPEN Open Source Project. This means that:
Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.
See the Contribution Guide for more details.
Donate
Support us with a monthly donation on Open Collective and help us continue our work.