ipfs-car-utils
v0.0.3
Published
Convert files to content-addressed archives (.car) and back
Downloads
71
Maintainers
Readme
ipfs-car 🚘✨⬢
Convert files to content-addressable archives (.car) and back
Description
ipfs-car
is a library and CLI tool to pack & unpack files from Content Addressable aRchives (CAR) file. A thin wrapper over @ipld/car and unix-fs.
Content-addressable archives store data as blocks (a sequence of bytes) each prefixed with the Content ID (CID) derived from the hash of the data; typically in a file with a .car
extension.
Use ipfs-car
to pack your files into a .car; a portable, verifiable, IPFS compatible archive.
$ ipfs-car --pack path/to/files --output my-files.car
or unpack files from a .car, and verify that every block matches it's CID
$ ipfs-car --unpack my-files.car --output path/to/write/to
Fetch and locally verify files from a IPFS gateway over http
curl -X POST "https://ipfs.io/api/v0/dag/export?arg=bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu" | 🚘
Install
# install it as a dependency
$ npm i ipfs-car
# or use the cli without installing via `npx`
$ npx ipfs-car --help
Usage
--pack
files into a .car
# write a content addressed archive to the current working dir.
$ ipfs-car --pack path/to/file/or/dir
# specify the car file name.
$ ipfs-car --pack path/to/files --output path/to/write/a.car
--unpack
files from a .car
# unpack files to a specific path.
$ ipfs-car --unpack path/to/my.car --output /path/to/unpack/files/to
# unpack specific roots
$ ipfs-car --unpack path/to/my.car --root <cid1> [--root <cid2>]
# unpack files from a .car on stdin.
$ cat path/to/my.car | ipfs-car --unpack
List the contents of a .car
# list the files.
$ ipfs-car --list path/to/my.car
# list the cid roots.
$ ipfs-car --list-roots path/to/my.car
# list the cids for all the blocks.
$ ipfs-car --list-cids path/to/my.car
API
To pack files into content-addressable archives, you can use the functions provided in:
ipfs-car/pack
for consuming a CAR writer async iterableipfs-car/pack/blob
for getting a blob with the CAR fileipfs-car/pack/fs
for storing in the local file system (Node.js only)ipfs-car/pack/stream
for writing to a writable stream (Node.js only)
⚠️ While packing files into CAR files, a Blockstore is used for temporary storage. All pack functions provide a default, but you can use other options (if supported on your runtime).
To unpack content-addressable archives to files, you can use the functions provided in:
ipfs-car/unpack
for getting an async iterable of the UnixFS entries stored in the CAR fileipfs-car/unpack/fs
for writing the unpacked files to disk (Node.js only)
ipfs-car/pack
Takes an ImportCandidateStream and returns a a CAR writer async iterable.
import { pack } from 'ipfs-car/pack'
import { MemoryBlockStore } from 'ipfs-car/blockstore/memory' // You can also use the `level-blockstore` module
const { root, out } = await pack({
input: [new Uint8Array([21, 31, 41])],
blockstore: new MemoryBlockStore(),
wrapWithDirectory: true // Wraps input into a directory. Defaults to `true`
maxChunkSize: 262144 // The maximum block size in bytes. Defaults to `262144`. Max safe value is < 1048576 (1MiB)
})
const carParts = []
for await (const part of out) {
carParts.push(part)
}
ipfs-car/pack/blob
Takes an ImportCandidateStream and writes it to a Blob.
import { packToBlob } from 'ipfs-car/pack/blob'
import { MemoryBlockStore } from 'ipfs-car/blockstore/memory' // You can also use the `level-blockstore` module
const { root, car } = await packToBlob({
input: [new Uint8Array([21, 31, 41])],
blockstore: new MemoryBlockStore()
})
ipfs-car/pack/fs
Takes a path on disk and writes it to CAR file (Node.js only).
import { packToFs } from 'ipfs-car/pack/fs'
import { FsBlockStore } from 'ipfs-car/blockstore/fs'
await packToFs({
input: `${process.cwd()}/path/to/files`,
output: `${process.cwd()}/output.car`,
blockstore: new FsBlockStore()
})
// output.car file now exists in process.cwd()
ipfs-car/pack/stream
Takes a writable stream and pipes the CAR Writer stream to it (Node.js only).
import fs from 'fs'
import { packToStream } from 'ipfs-car/pack/stream'
import { FsBlockStore } from 'ipfs-car/blockstore/fs'
const writable = fs.createWriteStream(`${process.cwd()}/output.car`)
await packToStream({
input: `${process.cwd()}/path/to/files`,
writable,
blockstore: new FsBlockStore()
})
// output.car file now exists in process.cwd()
ipfs-car/unpack
Takes a CAR reader and yields files to be consumed.
import fs from 'fs'
import { unpack } from 'ipfs-car/unpack'
const inStream = fs.createReadStream(`${process.cwd()}/output.car`)
const carReader = await CarReader.fromIterable(inStream)
const files = []
for await (const file of unpack(carReader)) {
// Iterate over files
}
Takes an AsyncIterable and yields files to be consumed.
import fs from 'fs'
import { unpackStream } from 'ipfs-car/unpack'
const inStream = fs.createReadStream(`${process.cwd()}/output.car`)
const files = []
for await (const file of unpackStream(inStream)) {
// Iterate over files
}
unpackStream
takes an options object, allowing you to pass in a BlockStore
implementation. The blocks are unpacked from the stream in the order they appear, which may not be the order needed to reassemble them into the Files and Directories they represent. The blockstore is used to store the blocks as they are consumed from the stream. Once the stream is consumed, the blockstore provides the random access by CID to the blocks, needed to assemble the tree.
The default is a MemoryBlockStore
, that will store all the blocks in memory.
For larger CARs in the browser you can use IndexedDB by passing in an IdbBlocksStore, and in Node.js you can provide a [FsBlockStore] instance to write blocks to the tmp dir.
/* browser */
import { unpackStream } from 'ipfs-car/unpack'
import { IdbBlockStore } from 'ipfs-car/blockstore/idb'
const res = fetch(
'https://ipfs.io/api/v0/dag/export?arg=bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy',
{ method: 'POST' }
)
const files = []
const blockstore = new IdbBlockStore()
for await (const file of unpackStream(res.body, { blockstore })) {
// Iterate over files
}
blockstore.destroy()
When providing a custom Blockstore, it is your responsibiltiy to call blockstore.destroy()
when you're finished. Failing to do so will fill up the users storage.
ipfs-car/unpack/fs
Takes a path to a CAR file on disk and unpacks it to a given path
import { unpackToFs } from 'ipfs-car/unpack/fs'
await unpackToFs({
input: `${process.cwd()}/my.car`,
output: `${process.cwd()}/foo`
})
// foo now exists in process.cwd()
// it is either a file or a directory depending on the contents of the .car
Takes a stream to a CAR file and unpacks it to a given path on disc
import fs from 'fs'
import { unpackStreamToFs } from 'ipfs-car/unpack/fs'
const input = fs.createReadStream(`${process.cwd()}/my.car`)
await unpackStreamToFs({
input,
output: `${process.cwd()}/foo`
})
// foo now exists in process.cwd()
// it is either a file or a directory depending on the contents of the .car