npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

stream-json

v1.9.0

Published

stream-json is the micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API. I

Downloads

6,472,511

Readme

stream-json NPM version

stream-json is a micro-library of node.js stream components with minimal dependencies for creating custom data processors oriented on processing huge JSON files while requiring a minimal memory footprint. It can parse JSON files far exceeding available memory. Even individual primitive data items (keys, strings, and numbers) can be streamed piece-wise. Streaming SAX-inspired event-based API is included as well.

Available components:

  • Streaming JSON Parser.
    • It produces a SAX-like token stream.
    • Optionally it can pack keys, strings, and numbers (controlled separately).
    • The main module provides helpers to create a parser.
  • Filters to edit a token stream:
    • Pick selects desired objects.
      • It can produces multiple top-level objects just like in JSON Streaming protocol.
      • Don't forget to use StreamValues when picking several subobjects!
    • Replace substitutes objects with a replacement.
    • Ignore removes objects.
    • Filter filters tokens maintaining stream's validity.
  • Streamers to produce a stream of JavaScript objects.
    • StreamValues can handle a stream of JSON objects.
      • Useful to stream objects selected by Pick, or generated by other means.
      • It supports JSON Streaming protocol, where individual values are separated semantically (like in "{}[]"), or with white spaces (like in "true 1 null").
    • StreamArray takes an array of objects and produces a stream of its components.
      • It streams array components individually taking care of assembling them automatically.
      • Created initially to deal with JSON files similar to Django-produced database dumps.
      • Only one top-level array per stream is valid!
    • StreamObject takes an object and produces a stream of its top-level properties.
      • Only one top-level object per stream is valid!
  • Essentials:
    • Assembler interprets a token stream creating JavaScript objects.
    • Disassembler produces a token stream from JavaScript objects.
    • Stringer converts a token stream back into a JSON text stream.
    • Emitter reads a token stream and emits each token as an event.
      • It can greatly simplify data processing.
  • Utilities:
    • emit() makes any stream component to emit tokens as events.
    • withParser() helps to create stream components with a parser.
    • Batch batches items into arrays to simplify their processing.
    • Verifier reads a stream and verifies that it is a valid JSON.
    • Utf8Stream sanitizes multibyte utf8 text input.
  • Special helpers:
    • JSONL AKA JSON Lines AKA NDJSON:
      • jsonl/Parser parses a JSONL file producing objects similar to StreamValues.
        • Useful when we know that individual items can fit in memory.
        • Generally it is faster than the equivalent combination of Parser({jsonStreaming: true}) + StreamValues.
      • jsonl/Stringer produces a JSONL file from a stream of JavaScript objects.
        • Generally it is faster than the equivalent combination of Disassembler + Stringer.

All components are meant to be building blocks to create flexible custom data processing pipelines. They can be extended and/or combined with custom code. They can be used together with stream-chain to simplify data processing.

This toolkit is distributed under New BSD license.

Introduction

const {chain}  = require('stream-chain');

const {parser} = require('stream-json');
const {pick}   = require('stream-json/filters/Pick');
const {ignore} = require('stream-json/filters/Ignore');
const {streamValues} = require('stream-json/streamers/StreamValues');

const fs   = require('fs');
const zlib = require('zlib');

const pipeline = chain([
  fs.createReadStream('sample.json.gz'),
  zlib.createGunzip(),
  parser(),
  pick({filter: 'data'}),
  ignore({filter: /\b_meta\b/i}),
  streamValues(),
  data => {
    const value = data.value;
    // keep data only for the accounting department
    return value && value.department === 'accounting' ? data : null;
  }
]);

let counter = 0;
pipeline.on('data', () => ++counter);
pipeline.on('end', () =>
  console.log(`The accounting department has ${counter} employees.`));

See the full documentation in Wiki.

Companion projects:

  • stream-csv-as-json streams huge CSV files in a format compatible with stream-json: rows as arrays of string values. If a header row is used, it can stream rows as objects with named fields.

Installation

npm install --save stream-json
# or: yarn add stream-json

Use

The whole library is organized as a set of small components, which can be combined to produce the most effective pipeline. All components are based on node.js streams, and events. They implement all required standard APIs. It is easy to add your own components to solve your unique tasks.

The code of all components is compact and simple. Please take a look at their source code to see how things are implemented, so you can produce your own components in no time.

Obviously, if a bug is found, or a way to simplify existing components, or new generic components are created, which can be reused in a variety of projects, don't hesitate to open a ticket, and/or create a pull request.

Release History

  • 1.9.0 fixed a slight deviation from the JSON standard. Thx Peter Burns.
  • 1.8.0 added an option to indicate/ignore JSONL errors. Thx, AK.
  • 1.7.5 fixed a stringer bug with ASCII control symbols. Thx, Kraicheck.
  • 1.7.4 updated dependency (stream-chain), bugfix: inconsistent object/array braces. Thx Xiao Li.
  • 1.7.3 added an assembler option to treat numbers as strings.
  • 1.7.2 added an error check for JSONL parsing. Thx Marc-Andre Boily.
  • 1.7.1 minor bugfix and improved error reporting.
  • 1.7.0 added utils/Utf8Stream to sanitize utf8 input, all parsers support it automatically. Thx john30 for the suggestion.
  • 1.6.1 the technical release, no need to upgrade.
  • 1.6.0 added jsonl/Parser and jsonl/Stringer.

The rest can be consulted in the project's wiki Release history.