npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

migrate-to-w3up

v1.0.3

Published

Migrate data from old web3.storage to w3up

Downloads

5

Readme

migrate-to-w3up ⁂

Migrate files to web3.storage.

Supported Migration Sources

  • https://old.web3.storage/
    • bring a WEB3_TOKEN environment variable from https://old.web3.storage/tokens/

Please file an issue to suggest a migration source that you would use.

Installation

npm install -g migrate-to-w3up

Usage

⚠️ Warning! ⚠️

  • if a migration errors part of the way through
    • you can't undo the partial migration
    • retrying the migration will start from the beginning, but anything already migrated should be super fast because it will just verify that the w3up space already has an Upload with the right cid.
  • this can migrate a lot of data rapidly! be sure you really want to add everything from the migration source to the destination.
    • to be extra careful, use filter mode and export your uploads from your source to a file. inspect the file to make sure it only contains uploads you want to migrate. Then pipe that file to migrate-to-w3up filter mode.
  • the space you migrate to must have a w3 storage provider attached. This usually means the storage provider will charge the account that added the storage provider to the space for any migrated files.

Easy mode (using the migration wizard 🧙‍♀️)

migrate-to-w3up

You will then see a series of prompts asking you:

  • where you want to migrate to, e.g. pick a w3up space
  • where you want to migrate from, e.g.
    • old.web3.storage uploads list from WEB3_TOKEN environment variable, if set and user confirms
    • old.web3.storage list from WEB3_TOKEN that user inputs into prompt

Advanced usage

Migrate w/ log

migrate-to-w3up --log /tmp/migrate-to-w3up.$(date +%s).log will run the migration and write migration events to the provided logfile. This includes UploadMigrationSuccess and UploadMigrationFailure events. The latter are logged along with the source upload that could not be migrated, and this allows the log file to serve as a source of uploads to be migrated in a second migration run to retry any failures.

# set this to a space did
MIGRATION_LOG_1="/tmp/migrate-to-w3up.$(date +%s).log"
migrate-to-w3up --log "$MIGRATION_LOG_1" --space "$W3_SPACE"
# wait quite some time
Retry Failures from Log

Because the log contains good records of any failures, you can use it to do a second migration run of any uploads that failed to migrate.

# MIGRATION_LOG_1 should be set from before
MIGRATION_LOG_2="/tmp/migrate-to-w3up.$(date +%s).log"

# retry migrating any uploads from UploadMigrationFailure
migrate-to-w3up log get-uploads-from-failures "$MIGRATION_LOG_1" | migrate-to-w3up --log "$MIGRATION_LOG_2" --space "$W3_SPACE"

# check for failures from second log. rinse and repeat?
migrate-to-w3up log get-uploads-from-failures "$MIGRATION_LOG_2"

Unix filter

You can also use migrate-to-w3up as a unix filter.

Just pipe in ndjson of old.web3.storage Uploads objects.

# gets current w3cli space (requires jq).
# copypasta from `w3 space ls` to pick another one,
# or do `w3 space use <space>` first
W3_SPACE=$(w3 space info --json | jq '.did' -r)

# define a command to get uploads as migration source
alias w32023-export='npx @web3-storage/w3@latest list --json'

migrate-to-w3up --space="$W3_SPACE" \
< <(w32023-export) \
| tee -a /tmp/migrate-to-w3up.$(date +%s).log
# include the previous line only if you want a logfile

# this also works
# jq optional but useful for pretty printing
# https://jqlang.github.io/jq/
w32023-export | migrate-to-w3up --space="$W3_SPACE" | jq

Migrate a single CAR part

Runs a single store/add invocation with the provided CAR link and show the output (this is mostly for w3up debuggers).

# base32 multihash CAR link
migrate-to-w3up $W3_SPACE store/add --link ciqgrph67ihh4imym4pl6d4xlnfhgxycdr4hcm6g6ucnzuxzqsorpsq

# CIDv1 CAR link
migrate-to-w3up $W3_SPACE store/add --link bagbaieranc6p56qopyqzqzy6x4hzow2konpqehdyoez4n5ie3tjptbe5c7fa

Example

$ migrate-to-w3up $W3_SPACE store/add --link ciqgrph67ihh4imym4pl6d4xlnfhgxycdr4hcm6g6ucnzuxzqsorpsq
{
  "ok": {
    "link": {
      "/": "bagbaieranc6p56qopyqzqzy6x4hzow2konpqehdyoez4n5ie3tjptbe5c7fa"
    },
    "with": "did:key:z6MkvSvQTdY87Rkazj3JiTquqQzjCAgWBuoWodPSHY6josAp",
    "status": "done",
    "allocated": 0
  }
}

How it Works

migrate-to-w3up/from-w32023

  • exports a migrate function that runs a migration, returning an AsyncIterable of MigratedUpload<W32023Upload> that includes ucanto receipts for every request sent to w3up as part of the migration.

migrate-to-w3up.js

  • this is the main bin script that runs when the package is globally installed and someone runs migrate-to-w3up
  • it reads cli flags to build options for migrate. If some options aren't provided as flags, and there is an interactive terminal, prompt the terminal for the options required to start a migration, e.g. selecting a source of uploads and destination (e.g. space DID).
  • it calls migrate from w32023-to-w3up.js and interprets the results as CLI output
    • it writes ndjson to stdout, and any errors/warnings to stderr.
    • it is recommended to pipe the output of this to a file so it can be explored later to check receipts or explore logs for debugging. But you can also pipe to jq to pretty print the ndjson output for ad-hoc exploration.