cloudQ

A http message/job queue that is easy to publish, consume and complete messages...

Install from source

git clone https://github.com/twilson63/node-cloudq.git
npm install .

# configure env vars
export COUCH=http://localhost:5984
export DB=cloudq
export TOKEN=foo
export SECRET=bar
export PORT=8000
export TIMEOUT=100000

npm run-script setup
npm start

Install and Run Locally

First, you need to install couchdb, you can download couchdb at http://couchdb.apache.org/

npm install cloudq -g
export COUCH=http://localhost:5984
export DB=cloudq
export TOKEN=foo
export SECRET=bar
export PORT=8000
export TIMEOUT=100000

# run server

cloudq

Usage

A job message queue server, allows your applications to push jobs to a queue, then worker applications can watch the queue and request for a job, when the worker receives the job, it does the work, then sends a complete message back to the server. Each job as a pre-defined schema that consists of two attributes:

• klass
• args

The klass attribute is a string represents the name of object that you wish to invoke. The args attribute is an array of parameters that you wish to provide to that objects perform method.

job schema

{ "job":
  {
    "klass": "Mailer",
    "args": [{"to": "[email protected]", "subject": "hello"}]
  },
  "priority": 100
}

publish

publishes the job to the queue named send_mail

curl -XPOST -d '{ "job": { "klass": "Mailer", "args": [{"to": "[email protected]", "subject": "hello"}]}}' \
-H "Content-Type: application/json" \
http://localhost:8000/send_mail

#> { "ok": true, "id": "c3b9e16d1efc436b7e30543bcf00182a", "rev": "1-977964400b51b5a1673fa7ac76d33874" } }

consume

consumes the next highest job in the queue

curl http://localhost:8000/send_mail
#>{ "klass": "Mailer", "args": [{"to": "[email protected]", "subject": "hello"}], "id": "1", "ok": true}

complete

curl -XDELETE http://cloudq.example.com/send_mail/1
#>{ "status": "success"}

Authorization

Currently authorization is done by environment varables:

TOKEN and SECRET

Theses env variables should match with basic authentication, per request:

curl http://token:secret@localhost:3000/foo

Test Successful Authentication:

curl -XPOST -d '{ "job": { "klass": "Mailer", "args": [{"to": "[email protected]", "subject": "hello"}]}}' http://token:[email protected]/send_mail

Logging

CloudQ uses bunyan as the logger and returns a stream of json, but if you want to put it into a more common format, then you can use the bunyan command to pipe the json into a readable format.

npm install bunyan -g
cloudq | bunyan

Produces:

2013-11-05T22:01:23.911Z]  INFO: cloudq/4187 on thing-4.local:
    0: {
      "ok": true,
      "id": "_design/dequeue",
      "rev": "17-d66392bf5441a2cae9bf4c52700cfeff"
    }
    --

for a shorter format

cloudq | bunyan -o short`

NewRelic

CloudQ is NewRelic Ready, simply supply an ENV Var for your New Relic key and you should be good to go.

# enable New Relic
export NEW_RELIC_LICENSE_KEY=xkkk
export NEW_RELIC_APP_NAME=cloudq 

cloudq | bunyan

Deploy

Deploy to nodejitsu

mkdir mycloudq
cd mycloudq
npm init
# edit package.json and set "node": "~0.6.x"
npm install cloudq --save
echo 'require("cloudq/server");' >> server.js
jitsu databases create couch cloudq
jitsu env set COUCH http://xxxx263878962530.iriscouch.com:5984
jitsu env set DB cloudq
jitsu env set TOKEN foo
jitsu env set SECRET bar

jitsu deploy

Deploy to heroku

# create an iriscouch account
mkdir mycloudq
cd mycloudq
npm init
# edit package.json and set "node": "~0.6.x"
npm install cloudq --save
echo 'web: ./node_modules/cloudq/bin/cloudq' >> Procfile
echo 'node_modules' >> .gitignore
git init
git add .
git commit -am "first commit"
heroku create
heroku config:add COUCH=http://mydb.iriscouch.com
heroku config:add DB=cloudq
heroku config:add TOKEN=foo
heroku config:add SECRET=bar

git push heroku master

Tests

npm test

License

see LICENSE

Contributing

GOALS

1. ONLY THREE CORE API METHODS

• POST /queue - PUBLISH a JOB on the QUEUE
• GET /queue - CONSUME a JOB
• DELETE /queue/id - Mark JOB as Completed

TODO

• tokens authorization
• create acl for queues, views, bulk updates

pull requests welcome