Github Downloader

Command-line utility to recursively download a directory from a repo on GitHub without needing to clone the entire project.

Installation:

npm install --global @warren-bank/node-github-downloader-cli

Methodology:

• GitHub REST API v3 is used to obtain metadata about the repo contents beginning at the desired path within the tree belonging to the desired commit
  • the tree is walked
    • directories are queried for additional metadata about the nested contents
    • files are downloaded to a path relative to the desired output directory

Usage:

ghdl <options>

options:
========
"-h"
"--help"
    Print a help message describing all "Github Downloader" command-line options.

"-V"
"--version"
    Display the version of "Github Downloader".

"-u <URL>"
"--url <URL>"
    Specify the URL to download.
    example:
      --url "https://github.com/torvalds/linux/tree/5908e6b738e3357af42c10e1183753c70a0117a9/Documentation/filesystems"
    where:
      * "torvalds"
          - is the name of the github user
      * "linux"
          - is the name of the repo
      * "5908e6b738e3357af42c10e1183753c70a0117a9"
          - is the commit SHA
      * "/Documentation/filesystems"
          - is the path to a directory within the repo (at the given commit)

"-P <dirpath>"
"--directory-prefix <dirpath>"
    Specifies the directory where all file downloads will be saved to.
    The default is "." (the current directory).

"-nc"
"--no-clobber"
    Indicates that a file download should not occur when the filepath to where it would be saved already exists.
    The default behavior is to delete the existing file and download the new file in its place.

"-o <filepath>"
"--output-file <filepath>"
    Specifies where to save a text file containing the list of all download file URLs,
    and corresponding relative output directory paths.
    The file format is compatible with "node-request-cli" (github.com/warren-bank/node-request-cli).
    Downloading of the file URLs does not occur.

Example:

• this test script is a good introduction

Requirements:

• Node.js version: v8.6.0 (and higher)
  • transitive dependency requirements:
    • v8.06.00+: @warren-bank/node-process-argv
    • v8.06.00+: @warren-bank/node-request

Other Similar Tools:

• github-files-fetcher
  • command-line
  • looks great
    • ..wish I found this tool before spending an evening writing my own
• DownGit
  • single-page web app
  • works great
• GitZip
  • single-page web app
• GitHub Folder Downloader
  • GUI desktop app
  • requires .NET 4.0+
• svn
  • GitHub still supports access to repos by subversion
  • ex:
    • directory in git repo:https://github.com/torvalds/linux/tree/master/Documentation/filesystems
    • access via subversion:svn checkout https://github.com/torvalds/linux/trunk/Documentation/filesystems
  • my humble opinion:
    • this seems like a massive amount of work for github's backend
    • support for this protocol is most likely going to eventually be removed
    • the day I go back to using svn (for any reason) will be the same day I reconnect a 56K modem to dialup

see:

• discussion on Stack Overflow

Legal:

• copyright: Warren Bank
• license: GPL-2.0