ISDP HSI download service

Installation

sudo yum install nodejs npm git

git clone [email protected]:hayashis/isdp.git
cd isdp
npm install

Upgrade

First, update the latest ISDP service

git pull

Then update ISDP dependencies..

npm update

Configuring

Copy ./config/config_sample.js to ./config/config.js

Edit ./config/config.js

See comments within the config file for more info, but if you aren't sure, feel free to contact me at [email protected]

Running

All production node apps should run under pm2. If you agree, install pm2

sudo npm install pm2 -g

Finally, start the ISDP service

pm2 start isdp.js

To auto-start pm2 (and isdp app) during the next reboot,

pm2 save
sudo pm2 startup redhat 

• Use "systemd" instead of redhat if you are installing on systemd enabled host.
• If you don't want to run the app as root, specify the user ID you want to run pm2 under (like -u hayashis)

Monitoring

pm2 stores stdout/stderr from ISDP to ~/.pm2/logs . you can monitor it by

pm2 logs isdp

You can check the status of ISDP server via /health endpoint

curl http://localhost:12346/health

If this doesn't return, or return non-200, then something is wrong.

You can also check the runtime information from pm2

pm2 show isdp

TODOs

1. Currently it receives request via web and immediately start processing request. This means all simultaneously requests will be executed in parallel... instead, I should do following.

• When a request comes, post to AMQP
• Write a separate handler that pulls request from AMQP one at a time and handle request

2. Currently doesn't handle invalid file path gracefully. Maybe I should skip missing ones and output a text file stating that files couldn't be downloaded?

3. Make sure an appropriate error message is generated (to who?) if incoming message is bogus.

4. Add a sensu check to make sure web-receiver and request handlers are running.

5. what happens if stage / publish directory can't be written? (doesn't exist, no access, not directory, disk is full)

6. add timeout mechanism for each task? also, should I async.retry?

7. what happens to error messages? stored in configured location (user can specify level)

8. there isn't much in the mocha test.. I need to add more unit testsing

• If caller send broken JSON, will it be logged in the error message?

DONE

9. what happens if amqp is down? I am not sure if it's pooling messages to be published locally, but as soon as amqp server comes back online, it reconnects seemslessly

10. what happens if 2 request comes in simultaneously? I am afraid "var job = this;" on async.series on sca-datamover will point to 2nd jobs while processing the 1st job? I've created a test script and it seems to work as expected. both job eventually finished successfully and correctly

What happens if unzip, tar, etc.. commands doesn't exists in path?