Node Backup (NPM, etc)

CLI for downloading public data (for now it is NPM packages meta or archives). Multithread.

Installation

npm install node-backup-loader -g

Usage

There is an important note. NPM has the only way to retrieve their packages, it is public search, therefore we must do about 1-2k requests to obtain information about all their packages, BUT, I found out it has much less packages in search, than actually can be found and installed directly (using https://www.npmjs.com/package/<package_name>)!

I found the repository called all-the-package-names which updated NPM packages list every day, and it has all of them (even deleted or deprecated).

So, there are two algorythms implemented to download NPM packages:

1. Fast and reliable, not all packages, search-way. Just get meta from NPM database using search, and download them. Bulk downloading (200 packages for each thread).

2. Slow, but you will receive all packages, package-api-way. You will request each packages one by one (they are not available on search API), so you will send about 1mm request, and it will produce huge load on NPM servers, your network connection, and, possibly, you hard drive (if you want to download archives).

Download packages meta

Meta data will be downloaded as JSON, one file for each package. At first, I implemented the way to download one huge JSON, but it become too large (about 5-10 Gb), so it was not possible to debug it or even open it. So the only way to download the data is to split it by files. Now all downloaded meta will be stored in the directory you specify. Total count of files is about 1mm, so if you are using fat32 table partition, create an issue then, you just will not be able to store all files in the same folder, restriction of fat32 systems is about 65k files per folder.

Commands:

• node-backup --command meta --registry npm --threads <threads_number> <directory> (fast-download, not all packages)
• node-backup --command meta --registry npm --npm-force-package-api --threads <threads_number> <directory> (slow download, all packages)

Examples:

• node-backup --command meta --registry npm --threads 25 ../my-folder/to/store/packages/meta/files/
• node-backup --command meta --registry npm --npm-force-package-api --threads 50 ../my-folder/to/store/packages/meta/files/

Download packages archives

Useful when you want to download all packages from NPM.

Commands:

• node-backup --command copy --registry npm [--preserve-downloaded-files] --threads <threads_number> <directory> (fast-download, not all packages)
• node-backup --command copy --registry npm [--preserve-downloaded-files] --npm-force-package-api --threads <threads_number> <directory> (slow download, all packages)

Examples:

• node-backup --command copy --registry npm --threads 1 ../my-folder/to/store/packages/archives/
• node-backup --command copy --registry npm --npm-force-package-api --threads 10 ../my-folder/to/store/packages/archives/
• node-backup --command copy --registry npm --preserve-downloaded-files --threads 1 ../my-folder/to/store/packages/archives/
• node-backup --command copy --registry npm --preserve-downloaded-files --npm-force-package-api --threads 10 ../my-folder/to/store/packages/archives/

I would not recommend you to set huge threads number (like, 100-200). For my case at any threads number more than 10-20, I could not get the downloading speed more than 5-15 packages per second due to npm pack command delay and my network connection speed.

The option --preserve-downloaded-files will force tool to not override already downloaded files in case if you accidentally closed your terminal. This options use the module glob, but as is there is too much files, it may cause performance issues, so it disabled by default.

Screenshots

How archives will be stored:

Downloading process:

Contribution

Look at the file src/modules/registry.ts and see what is registry. Every loaded data is abstracted from algorithm how it used, so you can create any new registry (like, bower, or any other database, from news blog to archive.org) and add new commands in this doc.

Be free to send pull requests.