tiny-async-pool

This is an ESM + TypeScript fork of tiny-async-pool by Rafael Xavier de Souza, with some added functionality.

Why?

The goal of this library is to use the native async iterator (ES9), native async functions, and native Promise to implement concurrent/batched work.

What?

doWorkAndYield runs multiple promise-returning & async functions in a limited concurrency pool. It rejects immediately as soon as one of the promises rejects. It calls the iterator function as soon as possible (under concurrency limit). It returns an async iterator that yields as soon as a promise completes (under concurrency limit). For example:

const timeout = (ms) => new Promise((resolve) => setTimeout(() => resolve(ms), ms));

for await (const ms of doWorkAndYield(2, [1000, 5000, 3000, 2000], timeout)) {
  console.log(ms);
}
// Call iterator timeout(1000)
// Call iterator timeout(5000)
// Concurrency limit of 2 reached, wait for the quicker one to complete...
// 1000 finishes
// for await...of outputs "1000"
// Call iterator timeout(3000)
// Concurrency limit of 2 reached, wait for the quicker one to complete...
// 3000 finishes
// for await...of outputs "3000"
// Call iterator timeout(2000)
// Itaration is complete, wait until running ones complete...
// 5000 finishes
// for await...of outputs "5000"
// 2000 finishes
// for await...of outputs "2000"

Usage

$ npm install @altano/tiny-async-pool

import { doWorkAndYield } from "@altano/tiny-async-pool";

ES9 for await...of

for await (const value of doWorkAndYield(concurrency, iterable, iteratorFn)) {
  console.log(value);
}

API

doWorkAndYield(concurrency, iterable, iteratorFn)

Runs multiple promise-returning & async functions in a limited concurrency pool. It rejects immediately as soon as one of the promises rejects. It calls the iterator function as soon as possible (under concurrency limit). It returns an async iterator that yields as soon as a promise completes (under concurrency limit).

concurrency

The concurrency limit number (>= 1).

iterable

An input iterable object, such as String, Array, TypedArray, Map, and Set.

iteratorFn

Iterator function that takes two arguments: the value of each iteration and the iterable object itself. The iterator function should either return a promise or be an async function.

doWork(concurrency, iterable, iteratorFn)

Like doWorkAndYield(...) but with a simpler API. Useful when you don't care about the return value of iteratorFn, such as a worker pool where you do batch file proccessing and the workers produce a side effect instead of returning a value.

For example, a batch file eraser is as simple as:

const eraseFile = (path) => fs.unlink(path);
const eraseFiles = async (files) => doWork(4, files, eraseFile);

It will be enforced that iteratorFn returns Promise<void> when using TypeScript.

License

MIT © Rafael Xavier de Souza MIT © Alan Norbauer