@itk-wasm/dam
v1.1.1
Published
Data Archive Manager
Downloads
4,417
Readme
@itk-wasm/dam
Data Archive Manager (dam): a tool to prepare and download source code repository testing, example, and website asset data.
Data is stored in .tar.gz archives and can be hosted at multiple, redundant https URLs. A simpler, more flexible alternative to Git LFS, etc.
Features
- Download and unpack testing and example data .tar.gz archives
- CLI for NPM scripts
- Verify with an expected hash compatible with web3.storage
- Cache archive downloads based on the expected hash
- Support for multiple storage urls
- Automatic retries
Installation
npm install -g @itk-wasm/dam
Usage
Store your data in a directory. For example:
> mkdir -p test/data
> touch test/data/file1.txt
> touch test/data/file2.txt
Pack the current version of the data into a compressed archive:
> dam pack test/data test/data.tar.gz
CID: bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
Upload the test/data.tar.gz file to hosting services such as web3.storage, GitHub Releases, AWS s3, a local http server, etc. For web3.storage, upload with the w3
CLI:
> npm install --global @web3-storage/w3
> w3 token
> w3 put --name dam-test-data --no-wrap test/data.tar.gz
# Packed 1 file (0.0MB)
# bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
⁂ Stored 1 file
⁂ https://w3s.link/ipfs/bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
Note the --no-wrap
flag.
In your NPM scripts, add
> dam download -v test/data test/data.tar.gz bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy https://w3s.link/ipfs/bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
If the data is stored at multiple locations, add all URLs to the end of the download
subcommand.
The first time this command is called the data will be downloaded and unpacked.
When data updates are required, repeat the pack / upload process, and update the content identifer (CID) hash.