Parallel Execution

Author: Neal Hu([email protected])

Description: A fault-tolerant NodeJS module to execute script in parallel

Documentation

execute(partitionFunc, partitionSpace, executionFunc, logDir, callback)

Execute executionFunc on each partition of partitionSpace, partitioned by partitionFunc. The status of the execution will be logged into logDir with a name 'task-xxxx' where xxxx is a timestamp of the start time of the execution.

partitionFunc is a function that takes partitionSpace and returns an array of partitions, which will be passed to executionFunc. executionFunc MUST be in the form executionFunc(partition, callback), and the callback function MUST be in the form callback(err, partition).

readFailedParitions(logFilePath, callback)

Read a execution log file specified in logFilePath, parse it and find all failed partitions. A completed partition is defined as a partition that is logged "started" and logged "completed" in the log. A failed partition is defined as a partition that is not completed. Error and an array of string presentation of failed partitions is passed to callback

Demo

This dummy example shows how to print an array in parallel. Define a partition space, a partition function and a execution function

var partitionSpace = [0, 1, 2, 3, 4, 5];
var partitionFunc = function(space) {
    var getModPartition = function(space, mod, total) {
        return space.filter(function(value, index) {
            return index % total == mod;
        })
    }
    return [0, 1, 2].map(function(v) {
        return getModPartition(space, v, 3);
    });
}

var executionFunc = function(partition, callback) {
    console.log("Executing on partition: " + partition);
    callback(null, partition);
}

This partition function will split the partition space [0, 1, 2, 3, 4, 5] into 3 partitions: [0, 3], [1, 4] and [2, 5]. The execution function will print each partition. Then you can call execution to execute the executionFunc on each partition in parallel.

var execution = new ParallelExecution();
execution.execute(partitionFunc, partitionSpace, executionFunc, '.', function(err, log) {
    execution.readFailedParitions(log, function(err, failed) {
        if (err) {
            console.log("Error: " + err);
        } else {
            console.log("Failed partitions: " + failed);
        }
    });
})

The status of the execution will be saved into '.' and then read back by readFailedParitions