bucket-maker

To reduce I/O cost, bucket-maker makes bucket of sliced timline for each unit you requested.
Name of the bucket is defined by its day and slot.

• day is a number of days from unix epoch
• slot is a number of unit from 00:00
• unit is defined by you

This is calculated like as follows:

unit = 5000;                                                // msec
days = Date.now() / (60*60*24*1000);                        // 17764.309366608795
day  = Math.floor(days);                                    // 17764
slot = Math.ceil( (days - day) * (60*60*24*1000) / unit );  // 5346

Demo

$ git clone https://github.com/RayKitajima/Bucket-Maker.git
$ cd Bucket-Maker

$ node demo/timestamp.js

demo/timestamp.js write out timestamp every sec into the bucket.
The bucket is made for every five sec.
So the bucket contains five entries in it.

At a grance

// get 5sec(msec) bucket maker
let bucket = require('bucket-maker').createBucket({
	unit: 5000
});

// put timestamp every sec
setInterval(function(){
	bucket.put(Date.now());
},1000);

// will write out
$ ls 
17764.4761.bucket  17764.4762.bucket

// will contain
$ cat 17764.4762.bucket
1534833405881
1534833406883
1534833407888
1534833408893
1534833409897

Options

Internally bucket is an array. And it will periodically dump and write them by dumper and writer.

| name | what is | |:----------|:--------------------------------------------------------------------| | prefix | prefix for output bucket name | | suffix | suffix for output bucket name. default <suffix> is '.bucket' | | unit | bucket unit as msec | | spool | where to write out bucket file | | dumper | how to dump elements in the bucket. by default assume them as stirng and join() them | | writer | how to write out dumped object. by default write out to file named as <prefix><day>.<slot><suffix>. | | start | whether to start writing at created | | cron_time | frequency of checking bucket | | log | how to system log |

Practice

const pid  = process.pid;
const host = os.hostname();
const prefix = pid + '.' + host + '.';

let bucket = require('bucket-maker').createBucket({
	prefix : prefix,
	suffix : '.bucket',
	unit   : 5000,
	spool  : '/path/to/buckets',
	dumper : function(items){ return items.map( x => JSON.stringify(x) ).join('\n') },
});

setInterval(function(){
	bucket.put(os.loadavg());
},1000);

Reporting load avarage every sec. And make 5sec bucket.

$ ls 
4825.localhost.17764.4761.bucket  4825.localhost.17764.4762.bucket

More reducing I/O cost with redis

const client = require('redis').createClient();

const writer = function(maker){
	let names = Object.keys(maker.Buckets);
	let now = Date.now();
	for( let i=0; i<names.length; i++ ){
		if( maker.Buckets[names[i]].cutoff > now ){ continue; }   // skip: still putting into the bucket
		let dump = maker.dumper(maker.Buckets[names[i]].buffer);
		let key  = name[i];                                       // wait:17764.4762.bucket
		client.set(key,dump,function(err,res){
			if( err ){
				maker.log(err);
			}else{
				delete maker.Buckets[names[i]];
			}
		});
	}
};

let bucket = require('bucket-maker').createBucket({
	prefix : 'wait:',
	unit   : 5000,
	dumper : function(items){ return items.map( x => JSON.stringify(x) ).join('\n') },
	writer : writer,
});

setInterval(function(){
	bucket.put(os.loadavg());
},1000);

This writes out loadavg report into redis.

const client = require('redis').createClient();

const drain = function(){
	client.keys('wait:*',function(err,keys){
		for( let i=0; i<keys.length; i++ ){
			client.get(keys[i],function(err,content){
				let bucket_name = keys[i].split(':')[1]; // 17764.4762.bucket
				fs.writeFileSync(bucket_name,content);
				client.del(keys[i]);
			});
		}
	});
};

setInterval(drain,5000);

This drains bucket to the file system.

If you would like to implement your own writer, you have to know about internal code, and should write several idiom to define what to write out.

License

MIT