progressive-downloader

A modern JavaScript / NodeJS library for parallel downloading of and verification of files using different hashing algorithms such as sha256, md5, and many more. Callback functions can be specified for reporting download progress and speed.

Usage

Install the library by running npm i progressive-downloader.

The download function is your one-stop-shop for everything this library provides. Specify an array of files. If you specify a hash for a file, it will be checked if it exists an not re-downloaded if the checksum matches. If the checksum does not match or the file does not exist, it will be re-downloaded and checked after download. Files specified without checksums will always be redownloaded to ensure they are still current. See the documentation for detailed API information.

const { download } = require("progressive-downloader");
download(
  [
    {
      url: "https://cdimage.ubports.com/something.txt",
      path: "/where/to/save/something.txt",
      checksum: {
        algorithm: "md5",
        sum: "d41d8cd98f00b204e9800998ecf8427e"
        // see https://nodejs.org/api/crypto.html#crypto_crypto_gethashes for supported hashes
      }
    },
    {
      url: "https://cdimage.ubports.com/something_else.txt",
      path: "/where/to/save/something_else.txt",
      checksum: {
        algorithm: "sha256",
        sum: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      }
    },
    {
      url: "https://cdimage.ubports.com/something_entirely_different.txt",
      path: "/where/to/save/something_entirely_different.txt"
      // if no checksum is specified, the file will always be re-downloaded
    }
  ],
  (progress, speed) => // called once per second while downloading
    console.log(`download ${progress * 100}% complete at ${speed} MB/s`),
  (current, total) => // called every time a file download completes
    console.log(`downloading file ${current} of ${total}`),
  (activity) => // "preparing" or "downloading"
    console.log(`running ${activity}`)
)
.then(() => console.log("download complete"))
.catch(error => console.error(`something terrible happened: ${error}`));

Documentation for the latest release available online. You can build it yourself by running npm run docs. Then, open ./docs/progressive-downloader/1.0.0/index.html in your favorite browser.

Development

$ npm install # to install dependencies
$ npm update # to update dependencies
$ npm audit fix # to install security patches
$ npm run lint # to check coding style
$ npm run lint-fix # to automatically fix coding style issues
$ npm run test # to run unit-tests with coverage reports
$ npm run test-nocover # to run unit-tests without coverage reports
$ npm run docs # to build detailed jsdoc documentation
$ npm run build # to compile backwards-compatible code using babel

Dependencies

Dependencies are managed using npm, the node package manager. Runtime dependencies are:

• @babel/runtime for language compatibility
• axios a widely used http library
• checksum wraps the native node hashing functions for convenient handling of files
• fs-extra wraps the native node fs module for convenient async integration

History

The library was originally developed for use in the UBports Installer, but it might be useful to other projects. Semantic versioning will ensure API stability for public functions.

License

Original development by Johannah Sprinz. Copyright (C) 2020-2021 UBports Foundation.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.