diego

Diego is a command line tool written with nodejs that is meant to make bulk downloading and extracting data a little bit easier. There is A lot of data out there that is annoying to collect and work with. Hopefully diego can save you some time and not get in your way too much when you have to download a bunch of data.

installation

First, you need to have nodejs installed.

To install diego js, in your terminal type: $ npm install -g diego

dependencies

commander, request, cheerio, async, adm-zip, ftp-get,

This has only been tested, so far, with node 4.1.x and node 0.12.x on osx 10.11.

tools

download tools

bulk file download

The download command opens your input URL, finds the file types you are looking for, and downloads them to your output directory.

    diego download [-f] [input url] [output directory]

optional

-f --fileTypes: The default download type is zip. If you want to download other filetypes include them here as a comma separated value. ex, to download all comma seperated, pdf, zip, and text files:

  diego download -f csv,pdf,zip,txt <inurl> <outfolder>

-n --nameLike: If you only want to download files with a specific string in the name, add it with this optional param. Example - You want to download only files with the word 'environment' in the name:

  diego download -n environment <inurl> <outfolder>

required

[input url]: This is the url you want to download from. Can be ftp, http, and https. Some https pages have issues.

[output directory]: This has to be an existing folder you have write permission to.

  diego download -f csv,xlsx,pdf,zip http://www.ecy.wa.gov/services/gis/data/data.htm ~/Downloads/census

  diego download http://www2.census.gov/geo/tiger/TIGER2015/TRACT/ ~/Downloads/census/

extracting

bulk file download

The extract command looks in your specified directory, and extracts all of the zip files.

coming soon - flatten tag, more compressed data types, delete original

    diego extract [directory]

required

[directory]: This has to be an existing folder you have write permission to which contains zip files.

  diego extract ~/Downloads/census

contact/questions/suggestions always welcome and appreciated