npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@aresrpg/sui-checkpoint-reader

v4.1.27

Published

<p align=center> <img src="https://user-images.githubusercontent.com/11330271/208825167-77d7bc78-17d0-4f33-ad35-d108b6fac730.gif" height="237px" width="344"/> </p> <h1 align=center>@aresrpg/sui-checkpoint-reader</h1> <p align=center> <img src="https:/

Downloads

311

Readme

This library allows building custom Sui indexers in JavaScript. It reads checkpoint files from both remote and local sources and processes them efficiently.

It also allows building a local leveldb from Sui's formal snapshots, which you can then process to initialize your indexer without the need to read checkpoints history for past epochs

Note: This is a work in progress and might not be suited for production usage.

🚀 Getting Started

First, install the library:

npm install @aresrpg/sui-checkpoint-reader

Here's a basic example of how to use the library:

import { read_checkpoints } from '@aresrpg/sui-checkpoint-reader/reader'

async function get_remote_checkpoint(num) {
  const response = await fetch(`https://checkpoints.testnet.sui.io/${num}.chk`)
  const buffer = await response.arrayBuffer()
  return buffer
}

async function process_checkpoint(checkpoint, index) {
  console.log('[indexer] process_checkpoint:', index)
}

const known_types = {}

await read_checkpoints({
  from: 1, // Start processing from checkpoint 1
  to: Infinity, // Continue processing indefinitely
  get_remote_checkpoint, // Function to fetch checkpoint data
  concurrent_downloads: 25, // Number of concurrent downloads allowed
  known_types, // BCS types you want to parse
  checkpoints_folder: '', // Local folder for checkpoint files (nesting is supported at 1 level)
  cleanup_checkpoints: false, // Clean up processed checkpoint files
  process_checkpoint, // Function to process each checkpoint
  local_files_only: false, // if true, the reader will only use checkpoints exported by the node
})

And here is how to get your leveldb from the formal snapshot

import { download_and_store_objects, read_snapshot_objects, get_db } from '@aresrpg/sui-checkpoint-reader/snapshot'

const DB_FOLDER = './sui-formal-objects'
const db = get_db(DB_FOLDER)

// this will start download the formal snapshot of the specified epoch
// and save any objects recognized by your known types into leveldb (key: id, value: object)
await download_and_store_objects({
  network: 'testnet',
  epoch: 440, // the epoch of the snapshot
  known_types, // same types as for checkpoint reading
  save_objects: false, // if you want to also save the .obj files locally under obj_folder/epoch_X/*.obj
  start_bucket = 1, // optional start files 1_1.obj
  start_part = 1,
  db,
  obj_folder = './obj_files' // if `save_objects` is true, it'll use this folder to save obj files
  concurrent_downloads = 1, // how many obj files to download in one go
})


// and here you can iterate easily on your leveldb, note that you can use anything else to read those
for await (const object of read_snapshot_objects(db)) {
  // process all objects here, for example use db.get(object.contents.kiosk_id)
  // to save something like { object, object_kiosk } in your storage of choice
  console.dir(object, { depth: Infinity })
}

⚙️ How It Works

The reader supports two modes: remote-only and hybrid (more powerful) mode.

Hybrid Mode (with checkpoints_folder specified)

  1. Initial Check: The reader will first check existing checkpoint files and determine what needs to be downloaded to fill the gap between from and the lowest present checkpoint file.
  2. Gap Filling: If from is lower than the lowest checkpoint file, missing checkpoints will be downloaded up to the lower checkpoint file.
  3. Sequential Processing: The reader will process checkpoints sequentially as they become available.
  4. Stopping Remote Reader: Once all checkpoints up to the lowest file (or the to parameter) are downloaded, the remote reader will stop.
  5. Local File Watching: Local checkpoint files will be watched and processed as they become available.

Remote-Only Mode (without checkpoints_folder)

In this mode, the reader will rely solely on remote checkpoint files:

  • Fetching Checkpoints: Checkpoints are fetched from the specified remote source.
  • Sequential Processing: Checkpoints are processed in sequential order as they are downloaded.
  • Efficient Downloads: The reader handles concurrent downloads up to the specified limit.

Example Usage for Remote-Only Mode

import { read_checkpoints } from '@aresrpg/sui-checkpoint-reader'

async function get_remote_checkpoint(num) {
  const response = await fetch(`https://checkpoints.testnet.sui.io/${num}.chk`)
  const buffer = await response.arrayBuffer()
  return buffer
}

async function process_checkpoint(checkpoint, index) {
  console.log('[indexer] process_checkpoint:', index)
}

const known_types = {}

await read_checkpoints({
  from: 1,
  get_remote_checkpoint,
  concurrent_downloads: 25,
  known_types,
  process_checkpoint,
})

🧩 BCS Types

To parse specific objects within the checkpoints, you can provide the BCS definitions directly.

Generating BCS Types

You can generate the necessary types using the https://github.com/Sceat/sui-bcs. Here is an example of how to generate them:

npx sui-bcs gen --package <package_address> --file <output_file> --network <network_name>

The network_name defaults to testnet if not provided. Not that you can also generate 0x2 types as needed

Example Usage

Here's an example of how you can define and use known BCS types:

// npx sui-bcs gen --package 0x68e06aee7966648c7b3258f02d5c7da2cf9b035727d6e9167c90e662f20e0406 --file aresrpg-bcs.js --network testnet
import { read_checkpoints } from '@aresrpg/sui-checkpoint-reader'

import aresrpg_bcs from './aresrpg-bcs.js'

const ARESRPG =
  '0x68e06aee7966648c7b3258f02d5c7da2cf9b035727d6e9167c90e662f20e0406'

const known_types = {
  [ARESRPG]: aresrpg_bcs,
}

This configuration allows the checkpoint reader to correctly parse and process the objects within the checkpoints.

Note that 0x1 & 0x2 are built in

📖 API Reference

read_checkpoints(options)

Options:

  • from: Start processing from this checkpoint number (default: 1).
  • to: Stop processing once this checkpoint number is reached (default: Infinity).
  • get_remote_checkpoint: Function to fetch a checkpoint array buffer from a number.
  • concurrent_downloads: Number of concurrent downloads allowed while catching up (default: 25).
  • known_types: Generated BCS for types you want to parse.
  • checkpoints_folder: Local folder where your Sui node is dumping the checkpoint files (default: '').
  • cleanup_checkpoints: Option to delete already processed checkpoint files (default: false).
  • process_checkpoint: Function to process a checkpoint.

download_and_store_objects(options)

Options:

  • network: Network to download the snapshot from (e.g., 'testnet').
  • epoch: Epoch of the snapshot.
  • known_types: Generated BCS for types you want to parse.
  • save_objects: If true, saves .obj files locally under obj_folder/epoch_X/\*.obj (default: false).
  • start_bucket: Optional start file (e.g., 1_1.obj) (default: 1).
  • start_part: Part to start from (default: 1).
  • db: Classic level instance.
  • obj_folder: Folder to store the .obj files
  • concurrent_downloads: Concurrent obj files download

read_snapshot_objects(db)

  • db: The Classic level db instance

Returns:

An async generator yielding [db, object] pairs for processing.

Contributing 🛠️

Contributions are welcome! Please open an issue or submit a pull request with any improvements or bug fixes.

License 📄

This project is licensed under the ISC License.