watch-putsftp

v1.0.4

Published

2 years ago

Watch a Folder and SFTP Put it to a server

Downloads

0High
0Medium
0Low

gitmft

mft watch sftp

Description

watch-putsfp is a smart JavaScript node.js sftp client CLI that listens for changes on local folders then uploads the file to an sFTP server.

It uses the node packages chokidar and ssh2 for file watching and sFTP file delivery respectively.

Requirements

node.js -- v0.10 or newer

Install

mkdir node_modules
npm install watch-putsftp

Run

cd node_modules/watch-putsftp
node index.js -u clarkkent -k sftpkey -h localhost -l /Volumes/WatchDisk/ -r uploads/
  Connecting to sFTP Server
  sftp: ready
  watcher: ready

Features

Watch local folders and upload to remote sFTP server
Auto retry if sFTP server goes down
Configurable watcher, polling, filters
Configurable sFTP client: password or keys, keep alive
Flexible CLI arguments utilizing yargs
Upload status with -s flag
sftp DEBUG option with -v 5 flag
Auto rename on server using .partial suffix

Configuration

Global config for this app

var myopts = {
  watchdir: '/tmp/mft/watch',  // -l flag
  remotedir: '',                 // -r flag
  partial: '.partial',         // write files with .partial suffix then rename
  deleteOnRename: true,        // delete original file if rename is used
  privatekey: '',
  verifyRemotedir: true,       // verify remotedir on db ready event
  conncnt: 0,
  connerr: '',
  conn: null,
  retryInterval: 30000         // retry 30 seconds if server goes down
};

Default config for ssh2

var sftpopts = {
  host: 'localhost',         // -h flag
  port: 22,                  // -p flag
  username: '',              // -u flag
  password: '',
  keepaliveInterval: 10000,
  keepaliveCountMax: 50,
  concurrency: 10,
  debug: sftpDebug,          // -v 5 flag
  // these settings are required for older products not supporting newer algos 
  algorithms: {
    kex: ['diffie-hellman-group1-sha1',
          'ecdh-sha2-nistp256',
          'ecdh-sha2-nistp384',
          'ecdh-sha2-nistp521',
          'diffie-hellman-group-exchange-sha256',
          'diffie-hellman-group14-sha1'],
    cipher: ['aes128-cbc',
          '3des-cbc','blowfish-cbc',
          'aes128-ctr','aes192-ctr',
          'aes256-ctr','aes128-gcm',
          '[email protected]',
          'aes256-gcm',
          '[email protected]'],
  },

  privateKey: ''            // -k flag
  };

Default config for chokidar

var watchopts = {
  ignored: /[\/\\]\./,    // ignore files atrting with '.'
  ignoreInitial: false,   // process existing files at start
  followSymlinks: false,
  persistent: true,
  depth: 0,               // ignore subdirs 
  awaitWriteFinish: { stabilityThreshold:2000, pollInterval: 100},
  alwaysStat: true,
  usePolling: true,       // use portable polling option
  interval: 100,
  binaryInterval: 300,
  ignorePermissionErrors: false,
  atomic: true
};

Client Examples

Display help and command line options

mode .
Usage: node index.js -putsftp.js: -u <SFTP_USER> -a <SFTP_PASSWORD> -k
<SFTP_KEY> -h <SFTP_HOST> -p <SFTP_PORT> -r <SFTP_FOLDER> -l <WATCH_FOLDER>  -v
<0-5> -s false

Options:
  -u, --user      Username for the remote SFTP Server.
  -a, --password  Password for the remote SFTP Server.
  -k, --key       SSH Key location for the remote SFTP Server.
  -h, --host      Host name or IP for the remote SFTP Server.         [required]
  -p, --port      Port for the remote SFTP Server.      [number] [default: 7522]
  -r, --remote    Remote folder where files are put for upload
  -v, --verbose   1-5 LOW to HIGH. Sends log items to the STDOUT console for
                  debugging purposes                       [number] [default: 1]
  -l, --local     Local watch folder where files are placed for upload[required]
  -s, --status    Provides updates on file upload status
                                                      [boolean] [default: false]
Examples:
  node watch-putsftp.js -u weblogic -a
  MyPassword -k /tmp/sftpkey -h localhost
  -p 7522 -r /uploads -l /tmp/mft/watch/
  -r /uploads/ -v 1 -s false

Listen on Disk WatchDisk and publish to localhost sFTP server with minimal auditing

bash-3.2$ node index.js -u clarkkent -k sftpkey -h localhost -r /uploads -l /Volumes/WatchDisk/ -r /clarkkent/uploads/
2016-03-25T21:48:35.976Z: Connecting to sFTP Server
2016-03-25T21:48:37.747Z: sftp: ready
2016-03-25T21:48:37.813Z: watcher ready: 
2016-03-25T21:49:14.953Z: upload sftp successful /Volumes/WatchDisk/package.json /clarkkent/uploads/package.json

Automatic error handling and retry when server goes down and comes back up

bash-3.2$ node index.js -u clarkkent -k sftpkey -h localhost -r /uploads -l /Volumes/WatchDisk/ -r /clarkkent/uploads/
2016-03-25T21:48:35.976Z: Connecting to sFTP Server
2016-03-25T21:48:37.747Z: sftp: ready
2016-03-25T21:48:37.813Z: watcher ready: 
2016-03-25T21:49:14.953Z: upload sftp successful /Volumes/WatchDisk/package.json /clarkkent/uploads/package.json
2016-03-25T21:51:05.173Z: SSH Client ERROR: Error: read ETIMEDOUT
2016-03-25T21:51:05.173Z: sFTP conn ERROR:ETIMEDOUT
2016-03-25T21:51:05.173Z: Pausing before reconnect 30 seconds 1
2016-03-25T21:51:05.174Z: sftp: close: 
2016-03-25T21:51:05.178Z: upload sftp invoke ERROR: /Volumes/WatchDisk/package.json Error: No response from server
2016-03-25T21:51:05.179Z: watcher upload ERROR: Error: No response from server
2016-03-25T21:51:36.487Z: sftp: ready
2016-03-25T21:51:36.500Z: watcher ready: 
2016-03-25T21:51:37.781Z: upload sftp successful /Volumes/WatchDisk/package.json /clarkkent/uploads/package.json

More verbose output with -v 3 flag

bash-3.2$ node index.js -u clarkkent -k sftpkey -h localhost -r /uploads -l /Volumes/WatchDisk/ -r /clarkkent/uploads/ -v 3
2016-03-25T22:01:50.918Z: Connecting to sFTP Server
2016-03-25T22:01:52.145Z: sftp: ready
2016-03-25T22:01:52.147Z: startWatching watcher init
2016-03-25T22:01:52.158Z: watcher addDir: /Volumes/WatchDisk/
2016-03-25T22:01:52.215Z: watcher ready: 
2016-03-25T22:02:01.171Z: watcher raw: change /Volumes/WatchDisk
2016-03-25T22:02:01.280Z: watcher raw: change /Volumes/WatchDisk/package.json
2016-03-25T22:02:03.200Z: watcher add file: /Volumes/WatchDisk/package.json
2016-03-25T22:02:03.201Z: upload entry: /Volumes/WatchDisk/package.json /clarkkent/uploads/ package.json package.json.partial
2016-03-25T22:02:03.501Z: upload sftp get stream resp:true
2016-03-25T22:02:03.964Z: upload sftp.fastPut: /Volumes/WatchDisk/package.json package.json.partial
2016-03-25T22:02:03.964Z: rename sftp entry: /clarkkent/uploads/package.json.partial /clarkkent/uploads/package.json
2016-03-25T22:02:03.964Z: unlink sftp entry: /clarkkent/uploads/package.json
2016-03-25T22:02:04.066Z: unlink sftp callback : /clarkkent/uploads/package.json
2016-03-25T22:02:04.067Z: unlink/delete file sftp successful rc: true /clarkkent/uploads/package.json
2016-03-25T22:02:04.067Z: rename unlink sftp successful: /clarkkent/uploads/package.json
2016-03-25T22:02:04.085Z: rename sftp callback : /clarkkent/uploads/package.json.partial /clarkkent/uploads/package.json
2016-03-25T22:02:04.086Z: rename sftp rc: true /clarkkent/uploads/package.json.partial /clarkkent/uploads/package.json
2016-03-25T22:02:04.087Z: upload sftp rename successful /clarkkent/uploads/package.json.partial /clarkkent/uploads/package.json
2016-03-25T22:02:04.087Z: upload sftp successful /Volumes/WatchDisk/package.json /clarkkent/uploads/package.json
2016-03-25T22:02:04.104Z: watcher raw: change /Volumes/WatchDisk/package.json
2016-03-25T22:02:04.177Z: watcher raw: change /Volumes/WatchDisk
2016-03-25T22:02:04.207Z: watcher unlink event: file deleted /Volumes/WatchDisk/package.json

Even more verbose output including SFTP DEBUG with -v 5 flag

bash-3.2$ node index.js -u clarkkent -k sftpkey -h localhost -r /uploads -l /Volumes/WatchDisk/ -r /clarkkent/uploads/ -v 5
2016-03-25T22:17:03.904Z: Connecting to sFTP Server
2016-03-25T22:17:03.929Z: sftpDebug: DEBUG: Local ident: 'SSH-2.0-ssh2js0.1.0'
2016-03-25T22:17:03.935Z: sftpDebug: DEBUG: Client: Trying 127.0.0.1 on port 22 ...
2016-03-25T22:17:04.033Z: sftpDebug: DEBUG: Client: Connected
2016-03-25T22:17:04.122Z: sftpDebug: DEBUG: Parser: IN_INIT
2016-03-25T22:17:04.125Z: sftpDebug: DEBUG: Parser: IN_GREETING
2016-03-25T22:17:04.126Z: sftpDebug: DEBUG: Parser: IN_HEADER
2016-03-25T22:17:04.127Z: sftpDebug: DEBUG: Remote ident: 'SSH-2.0-SSHD-CORE-0.6.0'
2016-03-25T22:17:04.130Z: sftpDebug: DEBUG: Outgoing: Writing KEXINIT
2016-03-25T22:17:04.133Z: sftpDebug: DEBUG: Parser: IN_PACKETBEFORE (expecting 8)
2016-03-25T22:17:04.133Z: sftpDebug: DEBUG: Parser: IN_PACKET
2016-03-25T22:17:04.134Z: sftpDebug: DEBUG: Parser: pktLen:268,padLen:14,remainLen:264
2016-03-25T22:17:04.134Z: sftpDebug: DEBUG: Parser: IN_PACKETDATA
2016-03-25T22:17:04.134Z: sftpDebug: DEBUG: Parser: IN_PACKETDATAAFTER, packet: KEXINIT
2016-03-25T22:17:04.140Z: sftpDebug: DEBUG: Comparing KEXINITs ...
...

File upload progress with -s flag

bash-3.2$ node index.js -u clarkkent -k sftpkey -h localhost -r /uploads -l /Volumes/WatchDisk/ -r /clarkkent/uploads/ -s
2016-03-25T22:21:11.128Z: Connecting to sFTP Server
2016-03-25T22:21:12.444Z: sftp: ready
2016-03-25T22:21:12.510Z: watcher ready: 
2016-03-25T22:21:19.589Z: Progress: /clarkkent/uploads/watch-putsftp.pdf.partial TotalTx:32768 Chunk:32768 Total:1363635
2016-03-25T22:21:19.680Z: Progress: /clarkkent/uploads/watch-putsftp.pdf.partial TotalTx:65536 Chunk:32768 Total:1363635
2016-03-25T22:21:19.780Z: Progress: /clarkkent/uploads/watch-putsftp.pdf.partial TotalTx:98304 Chunk:32768 Total:1363635
2016-03-25T22:21:19.858Z: Progress: /clarkkent/uploads/watch-putsftp.pdf.partial TotalTx:131072 Chunk:32768 Total:1363635
2016-03-25T22:21:19.875Z: Progress: /clarkkent/uploads/watch-putsftp.pdf.partial TotalTx:163840 Chunk:32768 Total:1363635

Tests

CLI tests in test folder ** test/sftpconn.js to test sftp connectivity ** test/sftplist.js to test file list ** test/sftpput.js to test file put

bash-3.2$ node test/sftpconn.js 
Usage: node sftpconn.js USERNAME PASSWORD KEYLOCATION HOSTNAME PORT
bash-3.2$ node test/sftplist.js 
Usage: node sftplist.js "." USERNAME PASSWORD KEYLOCATION HOSTNAME PORT
bash-3.2$ node test/sftpput.js 
Usage: node sftpput.js index.js "." USERNAME PASSWORD KEYLOCATION HOSTNAME PORT
bash-3.2$

TODO

Add pgp encryption
Notification via email and/or [node-notifier] (https://www.npmjs.com/package/node-notifier)
Add more formal tests
Propogate subfolders to server NOTE: it s very easy and light weigth to just start another instance
rename with timestamps/expressions
other?

Consider Contributing

Fork it!
Create your feature branch: git checkout -b my-new-feature
Commit your changes: git commit -am 'Add some feature'
Push to the branch: git push origin my-new-feature
Submit a pull request :D

History

Created: March 24, 2016

Credits

Dave Berry A.K.A (bigfiles)

License

ISC