nb-split-tasks

simple tool to split tasks then combine the result together for node or browser

This module is develop for the jsonql (especially in the contract-cli)

Notes

It only supports node.js at the moment (browser support coming soon)

As of 0.4.0 release, it support ES6 module, such as:

export default function add(x, y) {
  return x + y;
}

Please note, we are using the esm module to do the ES to CJS on the fly.

Checking your CPU before it runs

There is a little check to see how many CPU you have got, and the slots is number of cpu - 1 if it returns zero, that means you only have one processor (which is highly unlikely this day). Then it will return a rejected promise. You can check it like this:

const { NOT_ENOUGH_CPU } = require('nb-split-tasks/constants')
const nbSplitTasks = require('nb-split-tasks')

nbSplitTasks(pathToFn, args)
  .then(result => {
    // do your thing with the result
  })
  .catch(err => {
    if (err === NOT_ENOUGH_CPU) {
      // do alternative processing?
    }
  })

Too many file access might hang your machine

Internally, it's using the fork method to include your external file pathToFn; every time it split the task, it creates an entirely new instance of node.js, and I found if I try to access files from too many split tasks, it could hang the machine for a little while, therefore, use this with caution, even though my dev machine is core quad AMD with NVME SSD.

Examples

Just learn by examples

Return as an Array

By default it will merge the result as one object, or you can pass array as the third parameters then each result will put into an array

If you only have one function to call:

// fn.js
module.exports = function(x, y) {
  return x + y
}

Then

nbSplitTasks('/path/to/fn.js', args, 'array')
  .then(result => {
    // the return result will be
    // [3, 7, 11]
  })

If you have multiple functions then you can setup an array that is the same length as the arguments:

const nbSplitTasks = require('nb-split-tasks')

const pathToFns = [
  '/path/to/fn1.js',
  '/path/to/fn2.js',
  '/path/to/fn3.js'
] // or you can call the same methods with

const args = [
  [1,2],
  [3,4],
  [5,6]
]

// The function in each of the files
function fn1(x, y) {
  return x + y
}
function fn2(x, y) {
  return y - x;
}
function fn3(x, y) {
  return x * y
}
// don't forget to export that function as the default export

nbSplitTasks(pathToFns, args, 'array')
  .then(result => {
    // the return result will be
    // [3, 1, 30]
  })

Return as an Object

This is the default, and the function will try to merge all the result into one object; or you can pass an plain object with pre-defined properties, and let the result merge into it.

nbSplitTasks('/path/to/processor.js', args, {someKey: 'some init value'})
  .then(result => {
    assert(result.someKey, 'some init value')
  })

Cool little trick

This is from a live module I publish else where (not the exact code, but the idea is the same) If you only want to split the task out to another process, and don't want to create too many files. All you need is two

// the main.js
const nbSplitTasks = require('nb-split-tasks')

function doCal(x, y) {
  return x + y
}

function doExtralCal(value) {
  return value * 100
}

function callDoCal(x, y) {
  const pathToFn = './sub.js'
  return nbSplitTasks(pathToFn, [[x, y]]) // note array of array!
    .then(result => {
      // do a bit post processing if you want
      return result[0] // there is only result anyway
    })
}

module.exports = {
  doCal,
  doExtralCal,
  callDoCal
}

Next create the file for the callDoCal to use:

// sub.js
const { doCal, doExtralCal } = require('./main')
// of course this is overly simply in fact, the original
// main.js has several more methods that combine into one method
module.exports = function(x, y) {
  const result = doCal(x, y)
  return doExtralCal(result)
}

Then when you need it:

// app.js
const { callDoCal } = require('./main')

const result = callDoCal(1, 3)

Joel Zhu 2019 ISC