@iaigz/stream-io

Pipeable subprocesses

This is an utility to pipe to/from spawned child processes likewise a command line pipeline.

IOStream is responsible of ONE, and only one, Input-Output operation.

Internally, it wraps a Subprocess whoose standard input can be reached through the Writable stream interface; and whoose standard output can be consumed from the Readable stream counterpart.

[[TOC]]

Install

You may install this package either:

# using npm's gitlab support
npm install gitlab:iaigz/stream-io
# using npm.org public registry
npm install @iaigz/stream-io
# using gitlab.com @iaigz's npm package registry
npm config set @iaigz:registry=https://gitlab.com/api/v4/packages/npm/
npm install @iaigz/stream-io

Code example

const IOStream = require('@iaigz/stream-io')

// Ofc, we may pipe from any Readable source
process.stdin
  // Just pass data through
  .pipe(new IOStream('cat'))
  // Ofc, we may pipe to any Writable destination
  .pipe(process.stdout)

// WARN: favour usage of stream.pipeline() over Readable.pipe()
// see https://nodejs.org/docs/latest-v20.x/api/stream.html#streampipelinesource-transforms-destination-callback

For a more complete - but also useless - example, see example/jq.js

Test & Lint

A script is provided to run the suite. Once cloned, install deps and run it.

npm ci

npm test
# or
bash script/test
bash script/lint

Linting is done with standard, so code style meets standard style.

Knowledge Gems

The following tips are things I've found useful to understand streams better

Difference between Transform and Duplex

Time have elapsed since I messed-up with the Duplex aproach. Time ago I I thought IO may perform operations with data which aren't a transform - in the sense of transformation - but some day I found the following wonderful piece of knowledge at SO

Duplex Stream

A Duplex stream can be thought of a readable stream with a writable stream. Both are independent and each have separate internal buffer. The reads and writes events happen independently.

              ------------------|
        Read  <-----               External Source
You           ------------------|
        Write ----->               External Sink
              ------------------|
You don't get what you write. It is sent to another source.

Transform Stream

A Transform stream is a duplex where the read and write takes place in a causal way. The end points of the duplex stream are linked via some transform. Read requires write to have occurred.

               --------------|--------------
You     Write  ---->                   ---->  Read  You
               --------------|--------------
You write something, it is transformed, then you read something.

Conclusion

The EUREKA there is the linear flow input => output which a Transform stream eases significantly, whereas Duplex feels a kind of bi-directional beast.

That post has credits on why stream-io implements a Transform instead of a Duplex. It simplifies the whole thing, while technicaly an IO may not be writable at all (commands expecting no input) but produce readable output.

Backpressuring

There is a good guide about backpressuring at nodejs.org