datastore-zipcar (js-ds-zipcar)

An implementation of a Datastore for IPLD data that operates on ZIP files, supporting interacting via CID rather than the native Datastore Key type.

A Go implementation is also available at go-ds-zipcar.

Example

const ZipDatastore = require('datastore-zipcar')
const Block = require('@ipld/block')

async function example () {
  const block = Block.encoder(Buffer.from('random meaningless bytes'), 'raw')
  const cid = await block.cid()

  const ds = await ZipDatastore.readWriteFile('example.zcar')

  // store a new block, creates a new file entry in the ZIP archive
  await ds.put(cid, await block.encode())

  // retrieve a block, as a UInt8Array, reading from the ZIP archive
  const got = await ds.get(cid)

  console.log('Retrieved [%s] from zipcar with CID [%s]\n', Buffer.from(got).toString(), cid.toString())

  await ds.close()
}

example().catch((err) => {
  console.error(err)
  process.exit(1)
})

Will output:

Retrieved [random meaningless bytes] from zipcar with CID [bafkreihwkf6mtnjobdqrkiksr7qhp6tiiqywux64aylunbvmfhzeql2coa]

example.zcar is now a properly formatted ZIP archive:

$ unzip -l example.zcar
Archive:  example.zcar
  Length      Date    Time    Name
---------  ---------- -----   ----
       24  2019-08-12 04:15   bafkreihwkf6mtnjobdqrkiksr7qhp6tiiqywux64aylunbvmfhzeql2coa
---------                     -------
       24                     1 file

In this example, the readWriteFile() create-mode is used. This allows for read and mutation operations on a single ZipDatastore. Other create-modes are useful where the environment, data and needs demand:

• ZipDatastore.readWriteFile(file): read and mutate a ZIP archive file. Makes use of memory to cache reads and buffer the entire contents of the archive prior to writing. This mode is not available in a browser environment.
• ZipDatastore.readBuffer(buffer): read a ZIP archive from a Buffer or Uint8Array. Does not support mutation operations, only reads. This mode is the only mode available in a browser environment.
• ZipDatastore.readFile(file): read a ZIP archive directly from a file. Does not support mutation operations, only reads. However, this mode is very efficient for large data sets, with no caching and streaming reads internally. This mode is not available in a browser environment.
• ZipDatastore.writeStream(stream): write a ZIP archive to a stream (e.g. fs.createWriteStream(file)). Does not support read operations, only writes, and the writes are append-only (i.e. no delete()). However, this mode is very efficient for dumping large data sets, with minimal caching (the manifest at the end of the ZIP requires some caching of keys and positions), and streaming writes. This mode is not available in a browser environment.

The nature of ZIP archives (and the libraries currently available in JavaScript to work with them) means there are some awkward limitations. For example, there is no fromStream() operation because ZIP files require a seek to the end to read the manifest before entries may be individually read.

API

Contents

• async ZipDatastore.readBuffer(buffer)
• async ZipDatastore.readFile(file)
• async ZipDatastore.writeStream(stream)
• async ZipDatastore.readWriteFile(file)
• class ZipDatastore
• async ZipDatastore#get(key)
• async ZipDatastore#has(key)
• async ZipDatastore#put(key, value)
• async ZipDatastore#delete(key)
• async ZipDatastore#setRoots(comment)
• async ZipDatastore#getRoots()
• async ZipDatastore#close()
• async ZipDatastore#query([q])

async ZipDatastore.readBuffer(buffer)

Create a ZipDatastore from a Buffer containing the contents of an existing ZIP archive which contains IPLD data. The ZipDatastore returned will not support mutation operations (put(), delete(), setRoots()).

This create-mode is memory intensive as the Buffer is kept in memory while this ZipDatastore remains active. However, this create-mode is the only mode supported in a browser environment.

Parameters:

• buffer (Buffer|Uint8Array): the byte contents of a ZIP archive

Return value (ZipDatastore): a read-only ZipDatastore.

async ZipDatastore.readFile(file)

Create a ZipDatastore from an existing ZIP archive containing IPLD data. The ZipDatastore returned will not support mutation operations (put(), delete(), setRoots()) and reads will not perform any caching but be read via stream from the underlying file on demand.

This is an efficient create-mode, useful for reading the contents of an existing, large ZipDatastore archive.

This create-mode is not available in a browser environment.

Parameters:

• file (string): the path to an existing ZIP archive.

Return value (ZipDatastore): a read-only, streaming ZipDatastore.

async ZipDatastore.writeStream(stream)

Create a ZipDatastore that writes to a writable stream. The ZipDatastore returned will only support append operations (put() and setRoots(), but not delete()) and no caching will be performed, with entries written directly to the provided stream.

This is an efficient create-mode, useful for writing large amounts of data to ZIP archive.

This create-mode is not available in a browser environment.

Parameters:

• stream (WritableStream): a writable stream

Return value (ZipDatastore): an append-only, streaming ZipDatastore.

async ZipDatastore.readWriteFile(file)

Create a ZipDatastore from an existing ZIP archive containing IPLD data. The ZipDatastore returned will support both read operations (get(), has(), getRoots()), and mutation operations (put(),delete(), setRoots()). Reads will be cached in memory for as long as this ZipDatastore is active and a close(), where a mutation operation has been performed, will result in a full read and cache of the original archive prior to flushing back to disk.

This is an inefficient create-mode, useful for read/write operations on smaller data sets but not as useful for large data sets.

This create-mode is not available in a browser environment.

Parameters:

• file (string): the path to an existing ZIP archive.

Return value (ZipDatastore): a read-only, streaming ZipDatastore.

class ZipDatastore

ZipDatastore is a class to manage reading from, and writing to a ZIP archives using CIDs as keys and file names in the ZIP and binary block data as the file contents.

async ZipDatastore#get(key)

Retrieve a block from this archive. keys are converted to CID automatically, whether you provide a native Datastore Key object, a String or a CID. keys that cannot be converted will throw an error.

This operation may not be supported in some create-modes; a write-only mode may throw an error if unsupported.

Parameters:

• key (string|Key|CID): a CID or CID-convertable object to identify the block.

Return value (Buffer): the IPLD block data referenced by the CID.

async ZipDatastore#has(key)

Check whether a block exists in this archive. keys are converted to CID automatically, whether you provide a native Datastore Key object, a String or a CID. keys that cannot be converted will throw an error.

This operation may not be supported in some create-modes; a write-only mode may throw an error if unsupported.

Parameters:

• key (string|Key|CID): a CID or CID-convertable object to identify the block.

Return value (boolean): indicating whether the key exists in this Datastore.

async ZipDatastore#put(key, value)

Store a block in this archive. keys are converted to CID automatically, whether you provide a native Datastore Key object, a String or a CID. keys that cannot be converted will throw an error.

Depending on the create-mode of this ZipDatastore, the entry may not be written to the ZIP archive until close() is called and in the meantime be stored in memory. If you need to write a lot of data, ensure you are using a stream-writing create-mode.

Parameters:

• key (string|Key|CID): a CID or CID-convertable object to identify the value.
• value (Buffer|Uint8Array): an IPLD block matching the given key CID.

async ZipDatastore#delete(key)

Delete a block from this archive. keys are converted to CID automatically, whether you provide a native Datastore Key object, a String or a CID. keys that cannot be converted will throw an error.

If the key does not exist, delete() will silently return.

This operation may not be supported in some create-modes; a write-only mode may throw an error if unsupported. Where supported, this mode is likely to result in state stored in memory until the final close() is called.

Parameters:

• key (string|Key|CID): a CID or CID-convertable object to identify the block.

async ZipDatastore#setRoots(comment)

Set the list of roots in the ZipDatastore archive on this ZIP archive.

The roots will be written to the comment section of the ZIP archive when close() is called, in the meantime it is stored in memory.

In some create-modes this operation may not be supported. In read-only modes you cannot change the roots of a ZipDatastore and an error may be thrown.

Parameters:

• comment (string): an arbitrary comment to store in the ZIP archive.

async ZipDatastore#getRoots()

Get the list of roots set on this ZIP archive if they exist exists. See ZipDatastore#setRoots.

Return value (Array.<CID>): an array of CIDs

async ZipDatastore#close()

Close this archive, free resources and write its new contents if required and supported by the create-mode used.

If the create-mode of the current ZipDatastore supports writes and a mutation operation has been called on the open archive (put(), delete()), a new ZIP archive will be written with the mutated contents.

async ZipDatastore#query([q])

Create an async iterator for the entries of this ZipDatastore. Ideally for use with for await ... of to lazily iterate over the entries.

By default, each element returned by the iterator will be an object with a key property with the string CID of the entry and a value property with the binary data.

Supply { keysOnly: true } as an argument and the elements will only contain the keys, without needing to load the values from storage.

The filters parameter is also supported as per the Datastore interface.

Parameters:

• q (Object, optional): query parameters

Return value (AsyncIterator.<key, value>)

License and Copyright

Copyright 2019 Rod Vagg

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.