penny-collector

alpha warning

penny-collector is not ready for public consumption. it is published in order to facilitate a basic demo

install

• npm i -g penny-collector. Don't forget to update often via npm update -g penny-collector.
• penny-collector --save-config your-config-file.json
  • a custom config file can be tailored for your needs by the friendly folks at COINS
    • feel free to take a peek at the source in config/default.json if you'd like to make your own!
• @note: penny-collector require nodejs 4.x+

about

penny-collector is a data logging, submitting, and archiving utility.

penny-collector is designed to be run a system daemon. Contact your IT department for help on how to set this up. The COINS team can assist in this process. We can provide Ubuntu upstart script or help using a nodejs process monitor in setting up the daemon.

data submission

penny-collector can submit data in two modes, metadata mode and full-file mode.

metadata submissions

penny-collector can connect to COINS and submit information about research data. For example, study staff wish to submit records of "scan" images to COINS. They may also wish to submit "behavioral" data about their research participants. In this regard, staff may be able to associate and aggregate comprehensive data together. COINS can help study managers assess which participants have received which types of scans (where, when, how, etc), and query metadata about those events. Although in this mode the "raw" data is not available, the metadata is very often all that is required for research management.

full file submissions

penny-collector can transfer and archive data on the COINS system. This strategy is for users who want to use COINS as a storage service.

archiver

If you are not using penny-collector to submit files for storage at COINS, you may use penny-collector's archive functionality to move your files to archive storage based on rules. Provide a target.basePath and fields to compose your file paths from in target.fields. The fields will derive from your file meta-data, which we call a penny-model.

API Docs

Head over to the official docs to check out the API documentation.

ToDo

• [ ] add in real halfpenny validations!
• [ ] how to handle network failure (400 HTTP)
  • retry logic // e.g.: halfpenny.validateFileMeta([ ... array, of, file, meta ], { retry: true, retry: 3, retry: { attempts: 3, backoff: function() {} } })
  • lost and found
  • report (report API error response)
• how to handle api down (500s), same as 400s
• 403 Forbidden
  • what happens if validation loop is successful, same as others, but invalid
• 401 unauthorized,
  • process crash (via rethrow)

development

We recommend that, you, Ms. or Mr. Developer:

• add a local.json to config/

{
    "watchPath": ".test-watch",
    "target": {
        "basePath": ".test-target",
        "fields": ["ursi"]
    },
    "logFile": ".penny-collector.log"
}

appendix

#!upstart
description "COINS penny-collector"
author      "MRN-Code"

start on startup
stop on shutdown

script
    exec sudo -E -u your_user penny-collector
end script

credits

logo: margdking