multifeed-index
v3.4.2
Published
build an index over a set of hypercores
Downloads
101
Keywords
Readme
multifeed-index
Build an index over a set of hypercores.
Traverses a set of hypercores as a multifeed and calls a user indexing function to build an index.
Purpose
A multifeed is a set of append-only logs (feeds), with the property that only the feed's author can write new entries to it.
One type of document that you might write to such a feed are key -> value pairs. Maybe documents looks like this:
{
id: '23482934',
name: 'quinn',
gender: null
}
So some key, '23482934'
, can map to this document. How could you make
the lookup from '23482934'
to the aforementioned document fast? What if
there are thousands of such entries?
You'd probably want to build some kind of index. One that iterates over every entry in every feed, and also listens for new entries that get added. Then you could create an efficient data structure (say, maybe with level) that can map each key to a value quickly.
Good news: this is essentially what multifeed-index
does!
It does this by taking a multifeed you give it, along with three functions that the user provides:
storeState(state, cb)
: multifeed-index will give you aBuffer
object,state
, to store somewhere of your choosing. This could be in memory, to a database, to a JSON file, to browser storage, or whatever makes sense for your program. The module doesn't want to tie you down to a specific storage method.fetchState(cb)
: LikestoreState
, except this function will be called by multifeed-index when it needs to retrieve the state. Your job is to call the callbackcb
with the samestate
Buffer that was given tostoreState
earlier.clearIndex(cb)
: This only gets called when you change theversion
of the multifeed-index (see below). This function should delete the entire index and whateverstate
it stored earlier, so that the new version of the index can be regenerated from scratch.
Example
Let's build a key-value index using this and unordered-materialized-kv:
var multifeed = require('multifeed')
var indexer = require('multifeed-index')
var umkv = require('unordered-materialized-kv')
var ram = require('random-access-memory')
var memdb = require('memdb')
var multi = multifeed(ram, { valueEncoding: 'json' })
var kv = umkv(memdb())
var kvView = indexer({
version: 1, // setting a different number will cause the index to be purged and rebuilt
log: multi,
maxBatch: 5,
batch: function (nodes, next) {
var ops = nodes.map(function (node) {
return {
id: node.key.toString('hex') + '@' + node.seq,
key: node.value.key,
links: node.value.links
}
})
kv.batch(ops, next)
}
})
function append (w, data, cb) {
w.append(data, function (err) {
if (err) return cb(err)
var id = w.key.toString('hex') + '@' + (w.length - 1)
cb(null, id)
})
}
multi.writer(function (_, w) {
append(w, {
key: 'foo',
value: 'bax',
links: []
}, function (_, id1) {
console.log('id-1', id1)
append(w, {
key: 'foo',
value: 'bax',
links: [id1]
}, function (_, id2) {
console.log('id-2', id2)
kvView.ready(function () {
kv.get('foo', function (_, res) {
console.log(res)
})
})
})
})
})
outputs
id-1 9736a3ff7ae522ca80b7612fed5aefe8cfb40e0a43199174e47d78703abaa22f@0
id-2 9736a3ff7ae522ca80b7612fed5aefe8cfb40e0a43199174e47d78703abaa22f@1
[
'9736a3ff7ae522ca80b7612fed5aefe8cfb40e0a43199174e47d78703abaa22f@1'
]
API
var Index = require('multifeed-index')
var index = Index(opts)
Required opts
include:
log
: a multifeed instancebatch
: a mapping function, to be called on 1+ nodes at a time in the hypercores oflog
.storeState
: Function of the formfunction (state, cb)
. Called by the indexer when there is a new indexing state to save. The user can store this Buffer object whereever/however they'd like.fetchState
: Function of the formfunction (cb)
. Called by the indexer to seed the indexing process when the indexer is created. Expectscb
to be called with(err, state)
, wherestate
is a Buffer that was previously given toopts.storeState
.
The batch
function expects params (nodes, next)
. next
should be called
when you are done processing. Each node
of nodes
is of the form
{
key: 'hex-string-of-hypercore-public-key',
seq: 14,
value: <whatever value is in the hypercore at this feed + sequence number>
}
Optional opts
include:
version
: the version of the index. If increased, any indexes built with an earlier version will be purged and rebuilt. This is useful for when you change the data format of the index and want all peers to rebuild to use this format. Defaults to1
.maxBatch
: maximum batch size of nodes to process in onebatch
call. Defaults to50
.clearIndex
: Function of the formfunction (cb)
. Called by the indexer when a new version for the index has been passed in (viaopts.version
) and the index needs to be cleared & regenerated.
index.ready(cb)
Registers the callback cb()
to fire when the indexes have "caught up" to the
latest known change. The cb()
function fires exactly once. You may call
index.ready()
multiple times with different functions.
index.pause([cb])
Pauses the indexing process. Whatever batches of entries are currently being
processed will finish first. If a callback cb
is given, it will be called
once pending entries are processed and the indexer is fully paused.
index.resume()
Synchronously restarts a paused indexer.
var state = index.getState()
Returns the current state of the indexer.
state
is an object that looks like
{
state: 'indexing', // one of ['idle', 'indexing', 'paused', 'error']
context: {
totalBlocks: 50, // total # of blocks known of across all feeds
indexedBlocks: 18, // total # of blocks indexed so far
prevIndexedBlocks: 0, // total # of blocks indexed as of the previous indexing run
indexStartTime: 1583184761581, // ms since epoch when the last indexing run started
error: null // error state (indexing is unusable if this is set)
}
}
index.on('indexed', function (nodes) {})
Event emitted when entries have finished being indexed.
index.on('error', function (err) {})
Event emitted when an error within multifeed-index has occurred. This is very important to listen on, lest things suddenly seem to break and it's not immediately clear why.
index.on('state-update', function (state) {})
Event emitted when the internal state of the indexer changes. It has the same
form as the state
object returned by index.getState()
above.
Install
$ npm install multifeed-index
See Also
License
ISC