kpipe

This kpipe module combines the current versions of the main core modules of the kpipe data processing system. Including this module in a project will enable the project to use the base kpipe functionality without managing dependencies of the individual modules.

The modules included are:

| Module | Description | |---|--- | kpipe-core | Storage backends and basic readable and writable stream implementations. | | kpipe-streams | A set of low-level node transform streams to perform basic data conversion tasks such as compression, JSON translation, delineation, etc. | | kpipe-sequence | Provides an enhanced node pipeline running as a promise as well as tools to sequence a list of promisified actions. | | kpipe-url | Defines and supports a URL syntax for the kpipe stream backends. It automatically pipes together appropriate streams to deliver data in a particular format to/from a kpipe storage backend. |

kpipe-core

Support for reading and writing to storage backends using standard node streams.

Supported backends:

| Backend | Description | |---|---| | stdio | | | fs | | | s3 | | | kafka | | | buffer | | | random | |

| Function | Description | |---|---| | Reader | | | Writer | | | KafkaProducer | | | KafkaAdmin | |

kpipe-streams

Supported transforms:

| Transform | Description | |---|---| | Lineate | Add newlines to incoming strings and pack into a buffer | | Delineate | Split buffer into individual strings delimited by newlines | | JSONParse | Parse an incoming JSON string into a JavaScript object | | JSONStringify | Generate a JSON string from incoming JavaScripts objects | | Gzip | Compress a buffer stream using the zlib library | | Gunzip | Decompress a buffer stream using the zlib library | | SnappyCompress | Compress a buffer with snappy | | SnappyDecompress | Decompress a buffer with snappy |

kpipe-sequence

PipelinePromise

PromiseChain

kpipe-url

Translate URL-like paths into Kpipe backend stream generators.

| Pattern | Backend | Description | |---|---|---| | stdio:// - | stdio | | | fs://[path]/[file].[ext] file://[path]/[file].[ext] | fs | | | s3://[bucket](/[prefix])/[file].[ext] | s3 | | | kafka://[topic](/[partition](/[offset]))| kafka | |

| Function | Description | |---|---| | readerUrl | Return a Reader stream generator given a kpipe url | | readStreamUrl | Return a readable stream given a kpipe url | | writerUrl | Return a Writer stream generator given a kpipe url | | writeStreamUrl | Return a writable stream given a kpipe url | | urlInStreams | Given a kpipe url, return a readable stream which transforms backend data into the supplied content format (buffer, strings, or json) | | urlOutStreams | Given a kpipe url, return a writable stream which transforms data from the supplied content format (buffer, strings, or json) to a kpipe backend | | urlOutMultiplex | Special case of urlOutStreams which splits the incoming stream into one or more destination streams | | urlPipeline | Given an input URL, some transforms, and an output URL, return a promisified pipeline which performs a data conversion task | | urlPipelineSorted | Given an input URL, some transforms, and an output URL, return a promisified pipeline which first sorts the entire incoming dataset and then performs a data conversion task | | parseUrl | Helper function returns the parsed components of a kpipe url | | contentModifiers | Helper function returns an array of transforms to convert streams from one of the three content types (buffer, strings, or json) into another content type. | | compressExt | Helper function returns a transform stream (or passthrough) which compresses content according to a supplied filename extension | | decompressExt | Helper function returns a transform stream (or passthrough) which decompresses content according to a supplied filename extension |