npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

hdf5-indexed-reader

v1.0.1

Published

Module based on jsfive for efficient remote access of very large HDF5 files

Downloads

28

Readme

hdf5-indexed-reader

Summary

hdf-indexed-reader is a module for efficient querying of HDF5 files over the web. It enables loading of individual datasets from remote files without the need to load the entire file into memory. It works in conjunction with the companion project hdf5-indexer, which annotates HDF5 files with an index mapping object path names to file offsets.

The module is built on a fork of jsfive. The fork is available at https://github.com/jrobinso/hdf5-indexed-reader.

Motivation

The driving use case for this project involves extracting individual datasets for visualization in a web browser from large HDF5 files (~200 GB) containing 10s of thousands of individual datasets. Loading such files over the web with available solutions present 2 problems

  • The file is too large to load into browser memory in its entirety

  • Finding the file offset for the object desired involves walking a linked list of nodes of containing and sibling objects These nodes can be located anywhere in the file, resulting in an explosion of http range requests which can quickly freeze the application.

This project addresses these issues by (1) using range queries to load slices of the file as needed, and (2) supporting a pre-built index for mapping object (groups and datasets) paths to file offsets, negating the need to walking the linked list of container objects to build the index at runtime..

Limitations

  • As this project is based on jsfive, some limitations of that tool apply here, namely not all datatypes are supported.

  • This reader is designed for large HDF5 files containing many datasets. Small files will likely not
    benefit from indexing and incremental loading. Additionally, the benefit of indexing is reduced if the number of datasets is small.

Build

npm run install
npm run build

The build creates 3 packages

  • hdf5-indexed-reader.esm.js - an ES module for use in a web browser
  • hdf5-indexed-reader.node.cjs - a common JS module for use with Node
  • hdf5-indexed-reader.node.mjs - an ES module for use with Node

Usage

The module exports a single function, openH5File( {options} ). The HDF5 file is specified with one of the following properties

  • url - url to the hdf5 file
  • path - local file path, Node only
  • file - browser File or other Blob like object

URL fetches are cached to avoid separate individual requests for small amounts of data. The following optional properties controls the cache

  • fetchSize - minimum size in bytes for each http request. Defaults to 2000 (2 kb)
  • maxSize - the maximum number of bytes to cache. Default value is 200000 (200 kb)

In cases where it is not possible to modify the HDF5 file hdf5-indexer can create an external index as a json file. This file can be used with one of the following properties

  • indexURL - url to index json file
  • indexPath - local file path, node only
  • indexFile - browser File object

Example

Load a Dataset from a remote HDF5 file and fetch its shape, data type, and values.

import {openH5File} from "dist/esm/hdf5-indexed-reader.esm.js"

const hdfFile = await openH5File({
    url: "https://www.dropbox.com/s/53fbs3le4a65noq/spleen_1chr1rep.indexed.cndb?dl=0",
})

const spatialPostionDataset = await hdfFile.get('/replica10_chr1/spatial_position/1149')
const shape = await spatialPostionDataset.shape
const dtype = await spatialPostionDataset.dtype
const values = await spatialPostionDataset.value

See examples folder for node cjs and es examples.