npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

multi-core-indexer

v1.0.0

Published

Index one or more hypercores

Downloads

1,672

Readme

multi-core-indexer

Node.js CI Coverage Status Npm package version

⚠️ This is an Alpha release and the API will likely change. Do not use in production. ⚠️

Index one or more hypercores

You can use this module to index one or more hypercores. The batch() function will be called with every downloaded entry in the hypercore(s), and will be called as new entries are downloaded or appended. The index state is persisted, so indexing will continue where it left off between restarts.

This module is useful if you want to create an index of items in multiple Hypercores. There is no guarantee of ordering, so the indexer needs to be able to index unordered data. Sparse hypercores are supported, and undownloaded entries in the hypercore will be indexed when they are downloaded.

Table of Contents

Install

npm install multi-core-indexer

Usage

const MultiCoreIndexer = require('multi-core-indexer')
const raf = require('random-access-file')
const Hypercore = require('hypercore')

function createStorage(key) {
  return raf(`./${key}`)
}

function batch(entries) {
  for (const { key, block, index } of entries) {
    console.log(`Block ${index} of core ${key.toString('hex')} is ${block})
  }
}

const cores = [
  new Hypercore('./core1'),
  new Hypercore('./core2'),
]

const indexer = new MultiCoreIndex(cores, { storage: createStorage, batch })

API

const indexer = new MultiCoreIndexer(cores, opts)

cores

Required
Type: Array<Hypercore>

An array of Hypercores to index. All Hypercores must share the same value encoding (binary, utf-8 or json).

opts

Required
Type: object

opts.batch

Required
Type: (entries: Array<{ key: Buffer, block: Buffer | string | Json, index: number }>) => Promise<void>

Called with an array of entries as they are read from the hypercores. The next batch will not be called until batch() resolves. Entries will be queued (and batched) as fast as they can be read, up to opts.maxBatch. block is the block of data from the hypercore, key is the public key where the block is from, and index is the index of the block within the hypercore.

Note: Currently if batch throws an error, things will break, and the entries will still be persisted as indexed. This will be fixed in a later release.

opts.storage

Required
Type: (key: string) => RandomAccessStorage

A function that will be called with a hypercore public key as a hex-encoded string, that should return a random-access-storage instance. This is used to store the index state of each hypercore. (Index state is stored as a bitfield).

opts.reindex

Optional
Type: boolean

If true, the cores, and any new ones that are added, will be reindexed from scratch.

opts.maxBatch

Optional
Type: number

The max size of each batch in bytes.

opts.byteLength

Optional
Type: (entry: { key: Buffer, block: Buffer | string | Json, index: number }) => number

Optional function that calculates the byte size of input data. By default, if the value encoding of the underlying Hypercore is binary or utf-8, this will be the byte length of all the blocks in the batch. If the value encoding is json then this will be the number of entries in a batch.

indexer.state

Type: IndexState: { current: 'idle' | 'indexing' | 'closing' | 'closed', remaining: number, entriesPerSecond: number }

A getter that returns the current IndexState, the same as the value emitted by the index-state event. This getter is useful for checking the state of the indexer before it has emitted any events.

indexer.addCore(core)

core

Required
Type: Hypercore

Add a hypercore to the indexer. Must have the same value encoding as other hypercores already in the indexer.

Rejects if called after the indexer is closed.

indexer.idle()

Resolves when indexing state is 'idle'.

Resolves if the indexer is closed before this resolves. Rejects if called after the indexer is closed.

indexer.close()

Stop the indexer and flush index state to storage. This will not close the underlying storage - it is up to the consumer to do that.

No-op if called more than once.

indexer.unlink()

Unlink all index files.

This should only be called after close() has resolved, and rejects if not.

indexer.on('index-state', onState)

onState

Required
Type: (indexState: { current: 'idle' | 'indexing', remaining: number, entriesPerSecond: number }) => void

Event listener for the current indexing state. entriesPerSecond is the current average number of entries being processed per second. This is calculated as a moving average with a decay factor of 5. remaining is the number of entries remaining to index. To estimate time remaining in seconds, use remaining / entriesPerSecond.

indexer.on('indexing', handler)

handler

Required
Type: () => void

Event listener for when the indexer re-starts indexing (e.g. when unindexed blocks become available, either through an append or a download).

indexer.on('idle', handler)

handler

Required
Type: () => void

Event listener for when the indexer has completed indexing of available data. Note: During sync this can be emitted before sync is complete because the indexer has caught up with currently downloaded data, and the indexer will start indexing again as new data is downloaded.

Maintainers

@digidem

Contributing

PRs accepted.

Small note: If editing the README, please conform to the standard-readme specification.

License

MIT © 2022 Digital Democracy