npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

basic-cluster

v3.0.1

Published

Run multiple async tasks in parallel

Downloads

147

Readme

Basic Cluster

npm version codecov install size npm downloads License

A library for performing multiple tasks in parallel with control over resource usage.

Installation

npm install basic-cluster

Concepts

Cluster

The cluster controls the execution of multiple async tasks. It has a size which will control how many instances it can manage.

The cluster also accepts options for how it retries obtaining an instance to run submitted tasks. Retries are managed via exponential-backoff so check the docs over there for options.

Instance

An instance is simply an object managed by the cluster and acts as a context for tasks to run in. While running a task, an instance is considered busy and it cannot accept a new task until the current one completes.

Task

Task is what you submit to the cluster for it to run when possible.

Usage

Here's what you do:

  1. Create a Cluster.
  2. Submit tasks to it.
  3. When you are done, you can shutdown the cluster.

Basic usage

The BasicCluster class makes it easy to run a bunch of parallel tasks which don't depend on a managed instance object.

import { BasicCluster } from './src/cluster/BasicCluster';

const clusterSize = 3;
const cluster: BasicCluster = new BasicCluster(clusterSize);

const result = cluster.submit(async () => {
    // Do something
});

Shutdown

If you no longer require the cluster, you can shut it down. There are two options for this:

  1. Cluster#shutdown() will attempt to wait for any running task to complete before shutting down.
    1. After shutdown is requested, new task submissions are immediately rejected. Tasks waiting to be picked up will also be rejected.
    2. If there are running tasks, the cluster will retry to shutdown at a later time.
    3. When gracefully shutdown retries are exhausted, the cluster will forcefully shutdown.
  2. Cluster#shutdownNow() will forcefully shutdown the cluster, calling shutdown() on all its instances immediately.
    1. This does not cancel running tasks, so depending on how their built and what stage their in, they might still complete successfully.

Usage with instances

You can create clusters that use (potentially complex) instance objects, reusing them for new tasks. The example that inspired this package was a cluster of Puppeteer browser instances, which take some time to initiate and, as such, are a prime candidate for pooling.

To do so, you can implement the Instance interface and use the Cluster class directly with your instance.

SimpleInstance

The SimpleInstance is a utility for when you need an instance with some state and the shutdown is a no-op. Here's an example:

let i = 0;
const cluster: Cluster<SimpleInstance<number>> = new Cluster(3, () => new SimpleInstance(++i));
cluster.submit((instance) => {
    console.log(`Running task on instance ${instance.getValue()}`);
    // do something with the instance
});