Aliyun OSS Bridge CLI

Command-line-interface for uploading/downloading GeoTIFFs to TDX and Aliyun OSS.

Installation

You can install this package by using npm.

npm install @nqminds/aliyun-oss-bridge-cli

If you are checking out a development version, use lerna to install dependencies:

npx lerna bootstrap --scope @nqminds/aliyun-oss-bridge-cli --include-dependencies

Usage

You can either use npx aliyun-oss-bridge if you have npx installed, otherwise you can locate the binary in node_modules/.bin.

You can run npx aliyun-oss-bridge --help for help, or npx aliyun-oss-bridge <command> --help for help on a command.

$ aliyun-oss-bridge.js --help
aliyun-oss-bridge.js <command>

Commands:
  aliyun-oss-bridge.js upload               Uploads a file to the specified TDX
  <dataset-name> <filePath>                 dataset
  aliyun-oss-bridge.js download             Downloads a file from the given
  <dataset-name> <key> <filePath>           dataset.
  aliyun-oss-bridge.js list                 List all available datasets.

Options:
  --config         Load options and arguments from a JSON config file
                                            [default: .aliyun-oss-bridgerc.json]
  --tdx-id         The TDX ID to use for authentication      [string] [required]
  --tdx-secret     The TDX Secret to use for authentication  [string] [required]
  --aliyun-id      The Aliyun RAM Access ID to use for authentication
                                                             [string] [required]
  --aliyun-secret  The Aliyun RAM Access Secret to use for authentication
                                                             [string] [required]
  --tdx-folder-id  The id of the folder on the TDX  [string] [default: "2iQN6s"]
  --tdx-server     The url pointing to the TDX server
                                    [string] [default: "https://tdx.cropdoc.cn"]
  --tdx-ttl        The length of time (in seconds) your TDX login will be valid
                   for                       [number] [default: 3600 (One Hour)]
  --help           Show help                                           [boolean]
  --version        Show version number                                 [boolean]

Uploading a file

$ npx aliyun-oss-bridge upload --help
aliyun-oss-bridge upload <dataset-name> <filePath>

Uploads a file to the specified TDX dataset

Positionals:
  dataset-name  The name or id of the TDX dataset to upload to
                                                             [string] [required]
  filePath      The file to upload. Set to `-` to load from stdin
                                                             [string] [required]

Options:
  --config         Load options and arguments from a JSON config file
                                            [default: .aliyun-oss-bridgerc.json]
  --tdx-id         The TDX ID to use for authentication      [string] [required]
  --tdx-secret     The TDX Secret to use for authentication  [string] [required]
  --aliyun-id      The Aliyun RAM Access ID to use for authentication
                                                             [string] [required]
  --aliyun-secret  The Aliyun RAM Access Secret to use for authentication
                                                             [string] [required]
  --tdx-folder-id  The id of the folder on the TDX  [string] [default: "2iQN6s"]
  --tdx-server     The url pointing to the TDX server
                                    [string] [default: "https://tdx.cropdoc.cn"]
  --tdx-ttl        The length of time (in seconds) your TDX login will be valid
                   for                       [number] [default: 3600 (One Hour)]
  --help           Show help                                           [boolean]
  --version        Show version number                                 [boolean]
  --date           The date of the GeoTIFF as YYYY-MM-DD to store as metadata.
                                                                        [string]

Examples:
  aliyun-oss-bridge upload 'Potato       Uploads a GeoTiff file to the
  Planting Areas (O1)'                      dataset called 'Potato Planting
  /path/to/geotiff.tif --date 2019-12-25    Areas (O1). Adds the optional --date
                                            metadata of 25th December 2019.

You can use:

Options

You can store options in a JSON file, where each key is the name of the option. By default, the options will be loaded from a .aliyun-oss-bridgerc.json if it exists, but this can be overridden with the --config flag. For example:

{
  "tdx-id": "Your-tdx-sharekey-id",
  "tdx-secret": "your-tdx-sharekey-secret",
  "aliyun-id": "your-aliyun-RAM-account-id",
  "aliyun-secret": "your-aliyun-RAM-account-secret",
  "tdx-folder-id": "the-tdx-folder-that-stores-all-the-data",
  "tdx-server": "https://tdx.nqm-1.com"
}

Additionally, you can load options from environment variables, if they have the prefix ALIYUN_OSS_BRIDGE. For example, ALIYUN_OSS_BRIDGE_TDX_ID='Your-tdx-sharekey-id'.

Examples

For CropDoc MMU

Setup

Firstly, make sure that MMU's TDX sharekey is in the "Cropdoc MMU Importers" user group, so that they have the appropriate read/write permissions to access datasets.

Secondly, make sure that MMU's Aliyun RAM account access is in the "cropdoc-mmu" group. The permission policy is listed in @nqminds/cropdoc-schemas/system/oss-policy if you need to edit/recreate the group.

Finally, make a config JSON file that has the correct values for the TDX credentials, Aliyun RAM credentials, and the TDX CropDoc folder ID.

Usage

Installation

You can install this program by using npm.

Go into this directory and run

npm install @nqminds/aliyun-oss-bridge-cli

Config files

You should have access to two config files, which can be used with the --config flag:

• cropdoc-mmu-test-uk.json uses UK servers, and can be used for fast testing.
• cropdoc-mmu-china.json uses Chinese servers, and should be used for deployment.

List resources

To list the resources you have access to, run the following:

me@localhost: ~/aliyun-oss-bridge-cli $ npx aliyun-oss-bridge --config cropdoc-mmu-test-uk.json list
{
  '2iQN6s-potatoPlantingAreasO1': {
    id: '2iQN6s-potatoPlantingAreasO1',
    owner: '[email protected]/tdx.nqm-1.com',
    name: 'Potato Planting Areas (O1)'
  },
  '2iQN6s-diseaseOccurenceAreaO4': {
    id: '2iQN6s-diseaseOccurenceAreaO4',
    owner: '[email protected]/tdx.nqm-1.com',
    name: 'Disease Occurence Area (O4)'
  },
  '2iQN6s-blightOccurrenceAreaRO3': {
    id: '2iQN6s-blightOccurrenceAreaRO3',
    owner: '[email protected]/tdx.nqm-1.com',
    name: 'Late blight disease occurrence area R (O3)'
  },
  '2iQN6s-remoteSensingDataI1NasaGeotiff': {
    id: '2iQN6s-remoteSensingDataI1NasaGeotiff',
    owner: '[email protected]/tdx.nqm-1.com',
    name: 'Remote Sensing Data (I1)'
  }
}

Upload resources

You can upload GeoTIFF files to the above datasets using either their name or id:

me@localhost: ~/aliyun-oss-bridge-cli $ npx aliyun-oss-bridge --config cropdoc-mmu-test-uk.json upload '2iQN6s-blightOccurrenceAreaRO3' 1990-01-01.tif
Uploading key 1990-01-01.tif | ████████████████████████████████████████ | 100% || ETA: 0s'

Downloading resources

You can then download GeoTIFF files by specifying:

me@localhost: ~/aliyun-oss-bridge-cli $ npx aliyun-oss-bridge --config cropdoc-mmu-test-uk.json download '2iQN6s-blightOccurrenceAreaRO3' 1990-01-01.tif /tmp/path-to-download-to.tif
Download key 1990-01-01.tif | ████████████████████████████████████████ | 100% || ETA: 0s'