through2-batch
v1.1.1
Published
A stream that transforms chunks to batches form the stream
Downloads
4,961
Maintainers
Readme
through2-batch
A stream that transforms chunks to batches form the stream.
A way to use a Node.JS Transform stream that batches chunks into an array (default is 10). Objects/chunks will still come in the same order, just in batched arrays.
Built using through2 and has the
same API with the addition of a batchSize
option.
Non-objectMode
streams are supported for completeness.
Written by Nima Gardideh (halfmoon.ws) and used in production by Taplytics.
Install
npm install --save through2-batch
Examples
Process rows from a CSV in batches.
var through2Batch = require('through2-batch');
fs.createReadStream('data.csv')
.pipe(csv2())
.pipe(through2Batch.obj(
{batchSize: 100},
function (batch, enc, callback) {
var self = this;
console.log(batch.length); // 100
someThingAsync(batch, function (newChunk) {
self.push(newChunk);
});
}));
Don't specify a transform fn, and let the batches be processed by another stream
var through2Batch = require('through2-batch');
fs.createReadStream('data.csv')
.pipe(csv2())
.pipe(through2Batch.obj())
.pipe(getSomeOtherStreamProcessingBatches());
Contributing
Fixed or improved stuff? Great! Send me a pull request through GitHub or get in touch on Twitter @ngardideh.