npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@ipld/fbl

v3.0.1

Published

`Flexible Byte Layout` is an advanced layout for representing binary data.

Downloads

68

Readme

Flexible Byte Layout is an advanced layout for representing binary data.

It is flexible enough to support very small and very large (multi-block) binary data.

Usage

import * as codec from '@ipld/dag-cbor'
import { sha256 as hasher } from 'multiformats/hashes/sha2'
import { fromIterable } from '@ipld/fbl'
import fs from 'fs'

const stream = fs.createReadStream('path/to/file')

for await (const block of fromIterable(stream, { codec, hasher })) {
  storage.put(block)
}

API

fs.fromIterable(asyncIterable, { codec, hasher, algorithm=balanced() })

This method returns an async iterable of multiformats/block instances.

It accepts any async iterable, but the iterable must only yield instances of Buffer.

The algorithm is an async generator that takes an array of [ length, cid ] tuples and yields Block instances.

The default algorithm is for a balanced tree with a default limit of 1000 chunk references per block.

fs.size(buffer|block|decodedBlockData)

This method returns the size of a given FBL. It accepts either a buffer, Block instance or the data for an FBL root block.

fs.read(root, get, start=0, end=Infinity)

read returns and async generator that will yield Buffer instance for every chunk within the start and end boundaries.

root is a root block, CID, or decoded block for the root of the FBL tree.

get is async cid => Block(), and async function that takes a CID instance and returns a Block instance.

start and end are the offsets to slice out of the data. Any end offset larger than the total size of the FBL will read to the end of the FBL and finish without throwing an exception.

fs.balanced(limit=1000)

This method returns an async generator for a balanced tree with no more than limit part references per block.

Schema

Flexible Byte Layout is an advanced layout for representing binary data.

It is flexible enough to support very small and very large (multi-block) binary data.

type FlexibleByteLayout union {
  | Bytes bytes
  | NestedByteList list
  | &FlexibleByteLayout link
} representation kinded

type NestedByteList [ NestedByte ]

type NestedByte union {
  | Bytes bytes
  | NestedFBL list
} representation kinded

type NestedFBL struct {
  length Int
  part FlexibleByteLayout
} representation tuple

FlexibleByteLayout uses a potentially recursive union type. This allows you to build very large nested dags via NestedByteList that can themselves contain additional NestedByteLists, links to BytesUnions.

An implementation must define a custom function for reading ranges of binary data but once implemented, you can read data regardless of the layout algorithm used.

Since readers only need to concern themselves with implementing the read method, they do not need to understand the algorithms used to generate the layouts. This gives a lot of flexibility in the future to define new layout algorithms as necessary without needing to worry about updating prior impelementations.

The length property must be encoded with the proper byte length. If not encoded properly, readers will not be able to read properly. However, the property is not secure and a malicious encoder could write it as whatever they please. As such, it should not be relied upon when calculating usage against a quota or any similar calculation where there may be an incentive for an encoder to alter the length.