CodeEngine workers

This library is used inside CodeEngine to provide multi-threaded concurrency. It exports a WorkerPool class, which manages worker threads and uses them to process files.

NOTE: This is an internal library that is only intended to be used by CodeEngine. Using it outside of CodeEngine is discouraged.

WorkerPool class

This class creates worker threads, manages their lifecycle, and transfers CodeEngine files back and forth between threads for processing.

import WorkerPool from "@code-engine/workers";

// Create a new WorkerPool instance
let pool = new WorkerPool(engine);

try {
  // Import a FileProcessor plugin in all workers
  let processFile = await pool.importFileProcessor("./my-file-processor.js");

  // Process a file on one of the workers
  await processFile(myFile, run);
}
finally {
  // Safely dispose the pool and threads
  await pool.dispose();
}

WorkerPool constructor

The constructor accepts a CodeEngine object.

import WorkerPool from "@code-engine/workers";
import CodeEngine from "code-engine";

let engine = new CodeEngine();
let pool = new WorkerPool(engine);

WorkerPool.size

Read-only property that returns the number of worker threads in the pool. After the dispose() method is called, this property will always return zero.

import WorkerPool from "@code-engine/workers";
import CodeEngine from "code-engine";

let engine = new CodeEngine({ concurrency: 4 });
let pool = new WorkerPool(engine);
console.log(pool.size);   // 4

await pool.dispose();
console.log(pool.size);   // 0

WorkerPool.isDisposed

Indicates whether the dispose() method has been called. Once disposed, the WorkerPool instance is no longer usable.

import WorkerPool from "@code-engine/workers";

let pool = new WorkerPool(engine);
console.log(engine.isDisposed);     // false

await engine.dispose();
console.log(engine.isDisposed);     // true

WorkerPool.importFileProcessor(moduleId, [data])

Imports a CodeEngine FileProcessor plugin in all worker threads.

• moduleId: The module name or path. The module must export a FileProcessor function.

• data: (optional) Data to pass to the module. This is only relevant if the module's default export is a function accepts this data and returns a FileProcessor function.

import WorkerPool from "@code-engine/workers";
let pool = new WorkerPool(engine);

// Import a FileProcessor plugin in all workers
let processFile = await pool.importFileProcessor("./my-file-processor.js");

// Process a file on one of the workers
await processFile(myFile, run);

WorkerPool.importModule(moduleId, [data])

Imports a JavaScript module in all worker threads. The module export (if any) is ignored. This method is intended for loading polyfills, globals, hooks, and other modules with side-effects.

• moduleId: The module name or path

• data: (optional) Data to pass to the module. This is only relevant if the module's default export is a function that accepts this data.

import WorkerPool from "@code-engine/workers";
let pool = new WorkerPool(engine);

// Import a polyfill module in all worker threads
await pool.importModule("@babel/polyfill");

WorkerPool.dispose()

Terminates the worker threads and releases all system resources that are held by a WorkerPool instance. Once dispose() is called, the WorkerPool instance is no longer usable.

import WorkerPool from "@code-engine/workers";

let pool = new WorkerPool(engine);
await pool.dispose();

"error" event

This event is fired whenever an unhandled error occurs in any of the worker threads. If you don't handle this event, then Node.js will automatically terminate the process.

NOTE: When an unhandled error occurs, the WorkerPool instance and/or its worker threads may be left in an invalid or unusable state. For this reason, we recommend that you dispose the WorkerPool instance and stop using it.

import WorkerPool from "@code-engine/workers";
let pool = new WorkerPool(engine);

pool.on("error", (error) => {
  console.error("An unhandled error occurred:", error);
  pool.dispose();
});

Contributing

Contributions, enhancements, and bug-fixes are welcome! Open an issue on GitHub and submit a pull request.

Building

To build the project locally on your computer:

1. Clone this repo git clone https://github.com/CodeEngineOrg/code-engine-workers.git

2. Install dependencies npm install

3. Build the code npm run build

4. Run the tests npm test

License

@code-engine/workers is 100% free and open-source, under the MIT license. Use it however you want.

This package is Treeware. If you use it in production, then we ask that you buy the world a tree to thank us for our work. By contributing to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.

Big Thanks To

Thanks to these awesome companies for their support of Open Source developers ❤