npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

draccus

v1.1.2

Published

Writes messages from SQS to S3

Downloads

17

Readme

Draccus

Build Status

A tool for stashing messages queued up in Amazon's SQS.

Use it as a disaster recovery mechanism to drain a queue, or use it to guarantee delivery of transaction logs to persistent storage.

SQS Messages will only be deleted once they have been permanently stored, thus certain classed of errors may cause messages to be stored twice. This is by design to avoid the possibility of dropping messages. If you care about uniqueness make sure there is an identifier in the message body that can be used by any future batch processing jobs that might run over the messages.

Install

$ npm install draccus
$ npm test draccus

Usage

dracus.js --options_file path/to/config --queue_name some_queue_name

CLI Flags

  • --options_file : Path to a JSON file where options can be loaded from.
  • --access_key_id : AWS access key ID.
  • --secret_access_key : AWS secret access key.
  • --metadata_service_host : Hostname to use when looking up credentials via IAM (see below).
  • --queue_name : The name of the queue to read messages from.
  • --flush_frequency : How often to flush messages to the store. Default: 60s.
  • --s3_bucket : Name of an S3 bucket where messages should be written.
  • --out_dir : A local directory to write messages to (e.g. for dev).
  • --stdout : Simply logs messages to the console (warning: will still erase the queue contents).
  • --daemon : Whether the process should stay running once the queue is empty, and wait for further messages.
  • --filename_pattern : How to generate filenames, uses momentjs. for string formatting, in addition replacing PID with the process id. Default: X for unix timestamp.
  • --log_file : The path of a file to write logs too.
  • --log_raw_message : Specifies that the raw SQS message should be stored as JSON, instead of the message body.

--options_file

The above flags can be specified via a JSON file, along with additional AWS configuration options as described in the SDK Documentation.

{
  "accessKeyId": "YOURACCESSKEY",
  "secretAccessKey": "yoursecretkey",
  "sslEnabled": true,
  "region": "us-west-2",
  "queueName": "my_awesome_queue",
  "s3Bucket": "drained_queue",
  "filenamePattern": "YYYY/MM/DD/HH-X-PID"
}

The JSON flags are camel case equivalents of the CLI flags.

AWS Credentials

When running outside of EC2 you will need to pass in a valid access key and secret key. However, if you are running within EC2 and you omit the access key draccus will fall back to trying to use IAM based credentials, using the instance's role. See here for more details.

Throughput

Throughput should mostly be limited by network latency and SQS response time. From SQS documentation the theoretical limit per instance should be 500 messages per second.

In a test on an m1.small it took 40.7s to handle 10,000 messages. That is: receive, write to S3, and delete the message. The test flushed to S3 every 10s, resulting in 3x 100k files.

Test set up:

draccus-fill-queue --options_file aws.config --batches 1000
time draccus --options_file aws.config --flush_frequency 10

(If you use multiple workers remember to specify a different filename pattern or include PID in the filename pattern.)

Contributing

Questions, comments, bug reports, and pull requests are all welcome. Submit them at the project on GitHub. If you haven't contributed to a Medium project before please head over to the Open Source Project and fill out an OCLA (it should be pretty painless).

Bug reports that include steps-to-reproduce (including code) are the best. Even better, make them in the form of pull requests.

Author

Dan Pupius (personal website), supported by A Medium Corporation.

License

Copyright 2013 The Obvious Corporation.

Licensed under the Apache License, Version 2.0. See the top-level file LICENSE.txt and (http://www.apache.org/licenses/LICENSE-2.0).