s3-log-fetcher

Fetch logs from S3.

Features:

• incredibly fast
• log lines are pumped out as they come in
• incredibly fast
• uses {node,unix} streams
• incredibly fast

Installation

$ npm install s3-log-fetcher

Usage

const logFetcher = require('s3-log-fetcher')

const rs = logFetcher({
  from: '2015-10-03',
  until: '2015-11-04',
  bucket: 'sydney-my-app-test',
  key: '<S3 key>',
  secret: '<S3 secret>'
})

rs.pipe(stdout)

cli:

Usage: s3-log-fetcher [options]

Options:
  -h, --help      Output usage information
  -v, --version   Output version number
  -f, --from      Start date (required)
  -u, --until     End date (required)
  -k, --key       S3 access key, defaults to env.S3_ACCESS_KEY (required)
  -s, --secret    S3 secret key, defaults to env.S3_SECRET_KEY (required)
  -b, --bucket    S3 bucket name (required)

Examples:
  $ s3-log-fetcher -f 2015-10-03 -u 2015-10-03 \
    -b testbucket  # Fetch logs from S3 for 2015-10-03
  $ s3-log-fetcher -f 2015-10-03 -u 2015-10-13 \
    -b testbucket  # Fetch logs from S3 for 2015-10-03 until 2015-10-13

Docs: https://github.com/TabDigital/node-s3-log-fetcher
Bugs: https://github.com/TabDigital/node-s3-log-fetcher/issues

S3 directory structure

S3-log-fetcher expects the S3 bucket to follow a format. All files should either be prefixed by a date, or live in a directory that is prefixed with the date, this package don't care. For example:

. [ bucket: sydney-my-app-test ]
  - 2015-10-11-sydney-ec2-01.log
  - 2015-10-11-sydney-ec2-04.log
  - 2015-10-11
    - sydney-ec2-02.log
    - sydney-ec2-03.log
  - 2015-10-12
    - sydney-ec2-01.log
    - sydney-ec2-02.log
  - 2015-10-12-sydney-ec2-03.log
  - 2015-10-12-sydney-ec2-04.log

API

readStream = logFetcher.createReadStream(opts)

Create a readable stream for a start and end date. Takes the following opts:

• from (required): Start date.
• until (required): End date.
• key (required): S3 access key. Defaults to process.env.S3_ACCESS_KEY.
• secret (required): S3 secret key. Defaults to process.env.S3_SECRET_KEY.
• bucket (required): S3 bucket name.

License

MIT