npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

couch-statsd-backend

v0.1.5

Published

Backend for statsd to emit stats to couchdb

Downloads

14

Readme

couch-statsd-backend

Backend for StatsD, which publishes stats to CouchDB

Installation

Within your statsd install:

npm install couch-statsd-backend

Configuration

{

    port: 8125

    ,deleteIdleStats: true

    //, couchhost: '192.168.3.21'    //default: localhost
    //, couchport: '5984'            //default: 5984
    , couchdb: 'abtest'


    //all below are optional
    //, debug: true

    //,bulk_size: 2500      // (0 to disable)
    //,bulk_wait: 50

    //, id_prefix: "c0-"
    //, id_prefix: process.pid

    //choose just one
    //, id_generator: "couch"
    //, id_generator: "cuid"
    //, id_generator: "node-uuid-v1"
    //, id_generator: "node-uuid-v4"
    //, id_generator: "custom"
    //only needed for 'custom'
    //, uuid_generator: function(num){ return Math.round(new Date().getTime() / 1000) + '-' + num; }

      , backends: [ "./backends/couch" ]

}

Doc _id notes

id_prefix: Depending on the setup it may be useful to prefix the _id with a string to identify the statsd process which sent the doc. Default is not to add anything.

It is possible to choose how the couch doc._id is generated - by default a single request to couchhost/_uuids is made per flush but an _id can be generated with javascript instead.

If 'cuid' or 'node-uuid' are chosen you may need to first install the npms as they are not installed by default.

npm install cuid

and

npm install node-uuid

If custom is chosen then uuid_generator must also be set eg:

  , id_generator: "custom"
  , uuid_generator: function(num){ return Math.round(new Date().getTime() / 1000) + '-' + num; }  

Note: The uuid_generator function is passed 'num' which is incremented each call within a flush, this is reset on each flush.

###Bulking

bulk_size: Number of docs to submit per bulk request. To disable bulking set this to 0.

bulk_wait: Time to wait (ms) between sending each bulk of docs for a flush to couch. Be careful with this if total time to insert all docs per flush > time period of flush then docs to be posted will just keep growing.

It should be ok for the total time to insert all the bulks to run over the statsd flush period if a sudden rush of metrics arrive once in a while but if this keeps happening i suspect node will run out of memory to hold the backlog.

##Supported Metrics

  • Counters
  • Counter Rates
  • Gauges
  • Timers
  • Timer Data
  • Sets

###Counters & Counter Rates

Counters and counter rates are combined into one doc - this can be easily modified to create a separate docs for the counter_rates by un-commenting the a few lines.

echo "myservice.mycount:67|c" | nc -n -w 1 -u 192.168.3.22 8125 

Will add a doc like:

{
   "_id": "f83a62b3456f4c2451cb7a166100f3e8",
   "_rev": "1-d68546bae9169fa4b4a72b3ffe4167b9",
   "type": "counter",
   "name": "myservice.mycount",
   "count": 67,
   "rate": 6.7,
   "ts": 1424818189
}

If two counts of the same name within same flush period

echo "myservice.mycount:20|c" | nc -n -w 1 -u 192.168.3.22 8125 
echo "myservice.mycount:30|c" | nc -n -w 1 -u 192.168.3.22 8125 

Will add a doc like:

{
   "_id": "f83a62b3456f4c2451cb7a1661010a10",
   "_rev": "1-89dc6493c2418acb351e9767e6653c65",
   "type": "counter",
   "name": "myservice.mycount",
   "count": 50,
   "rate": 5,
   "ts": 1424818219
}

###Gauges

echo "myservice.mygauge:42|g" | nc -n -w 1 -u 192.168.3.22 8125 

Will add a doc like:

{
   "_id": "f83a62b3456f4c2451cb7a1661012b57",
   "_rev": "1-390bc981b63e2ca16d2ad42c042f9c1b",
   "type": "gauge",
   "name": "myservice.mygauge",
   "value": 42,
   "ts": 1424819100
}

###Timers & Timer Data

Timers and timer data are combined into one doc - this can be easily modified to create a separate docs for the timer_data by un-commenting the a few lines.

echo "myservice.mytimer:231|ms" | nc -n -w 1 -u 192.168.3.22 8125

Will add a doc like:

{
   "_id": "2d0f912baefa5e8c6e131e29c4482e67",
   "_rev": "1-30071b20880b8f0139a403856e0365d9",
   "type": "timers",
   "name": "myservice.mytimer",
   "durations": [
       231
   ],
   "mean_90": 231,
   "upper_90": 231,
   "sum_90": 231,
   "std": 0,
   "upper": 231,
   "lower": 231,
   "count": 1,
   "count_ps": 0.1,
   "sum": 231,
   "mean": 231,
   "median": 231,
   "ts": 1425055422
}

If more than one timer is received withing the same flush interval eg:

echo "myservice.mytimer:30|ms" | nc -n -w 1 -u 192.168.3.22 8125
echo "myservice.mytimer:23|ms" | nc -n -w 1 -u 192.168.3.22 8125
echo "myservice.mytimer:51|ms" | nc -n -w 1 -u 192.168.3.22 8125

Will add a doc like:

{
   "_id": "2d0f912baefa5e8c6e131e29c44835d1",
   "_rev": "1-5786babae3d5c82d6511a5ee7770fd86",
   "type": "timers",
   "name": "myservice.mytimer",
   "durations": [
       23,
       30,
       51
   ],
   "mean_90": 34.666666666666664,
   "upper_90": 51,
   "sum_90": 104,
   "std": 11.897712198383164,
   "upper": 51,
   "lower": 23,
   "count": 3,
   "count_ps": 0.3,
   "sum": 104,
   "mean": 34.666666666666664,
   "median": 30,
   "ts": 1425055602
}

###Sets

echo "myservice.myset:129|s" | nc -n -w 1 -u 192.168.3.22 8125 

Will add a doc like:

{
   "_id": "f83a62b3456f4c2451cb7a16617f1d84",
   "_rev": "1-07926b51ef9b5a163a9bde8f3635223c",
   "type": "set",
   "name": "myservice.myset",
   "set": {
       "129": "129"
   },
   "ts": 1424819898
}

*I am not sure if the set data structure is correct?

##Notes/Future Changes A test db with snappy compression of 2 million docs consumed 0.5Gb of space.

I suspect having the full field names in each document increases the storage requirements so may be better to save the metric field names as single letters for example for a 'counter' type:

{
   "_id": "f83a62b3456f4c2451cb7a1661010a10",
   "_rev": "1-89dc6493c2418acb351e9767e6653c65",
   "t": "c",
   "n": "myservice.mycount",
   "c": 50,
   "r": 5,
   "ts": 1424818219
}

where:

t = type  (where c = counter, g = gauge, t = timer and s = set)
n = name
c = count
r = rate

It may even be worth encoding the 'type' within the '_id' ?

Also on testing i did hit issues when adding more that 20k documents every 3 seconds - not 100% sure where the bottle neck was. Look at sending each bulk batch off to a different couchdb server in a round robin fasion so each couchdb doesnt need to handle all metrics from a single flush - this may not be needed with couch 2.0 clustering stuff :).