npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

json-anystream

v2.0.1

Published

Stream JSON objects no matter what the source is.

Downloads

110

Readme

json-anystream

Tests GitHub package version standard-readme compliant

Takes any stream that provides JSON objects (array of objects or single object) or newline delimited JSON objects (NDJSON) and turns it into a JSON objects emitting stream.

This module is motivated by a need to read a set of JSON objects, which can be given in different forms. The module

  • provide a stream that emits single JSON objects via the data event.
  • supports JSON objects, JSON arrays of objects, NDJSON, as well as mutlipart/form-data with a file containing one of those.
  • supports reading from local files (using fs). Content type is determined via file ending.
  • supports reading from URLs (using http/https). Content type is determined via Content-Type header or file ending.
  • provides an express middleware that wraps the above functionality and adds the stream as res.anystream.

Table of Contents

Install

npm i json-anystream

json-anystream v2.0+ requires Node.js 18 and is ES module only. For older versions or CommonJS, use v1.1.1.

Usage

Import

import * as anystream from "json-anystream"

anystream.make

async function make(input, type, adjust)

Takes an input (URL, file path, or existing stream) and makes a JSON object stream out of it. type can have one of the values "json", "ndjson", or "multipart", and is optional if the type can be inferred from the file ending or content type header. adjust is an optional adjustment method that is called for each object in the resulting stream before it is emitted.

Example:

const stream = await anystream.make("https://example.com/test.json")
stream.on("data", object => {
  // object contains single JSON objects
})
stream.on("end", () => {
  // stream is done
})
stream.on("error", error => {
  // an error occurred, see below for error types
})

You can also use for async:

const stream = await anystream.make("https://example.com/api/test") // assuming content type header is set
for await (let object of stream) {
  // object contains single JSON objects
}

For existing streams, anystream.make as simply a wrapper over anystream.convert. For URLs, http.get or https.get are used and the resulting stream is wrapped by anystream.convert. For files, fs.createReadStream is used and the resulting stream is wrapped by anystream.convert.

anystream.convert

async function convert(stream, type, adjust)

Takes an existing stream and returns a JSON object stream. The type parameter is required and is determined by the data of the stream:

  • json: Any JSON stream (single object or array of object) that is compatible with stream-json.
  • ndjson: Any stream of JSON objects compatible with ndjson.
  • multipart: A request object containing multipart/form-data data that contains the field data with a compatible .json or .ndjson file (busboy is used for parsing the form data).

adjust is an optional adjustment method that is called for each object in the resulting stream before it is emitted.

anystream.StreamAnyObject

A custom streamer for stream-json that takes a stream of either a single JSON object or an array of JSON objects and transforms it into a stream that emits only JSON objects. I.e. if the input stream contains a single JSON object, that object is assembled and then emitted as a whole, if the input stream contains an array of JSON objects, the objects are emitted as direct JSON objects.

import streamJson from "stream-json"
const parser = streamJson.parser
const pipeline = stream.pipe(parser()).pipe(new anystream.StreamAnyObject())
pipeline.on("data", object => {
  // ...
})
pipeline.on("end", () => {
  // ...
})

There is also a custom event in case the input was in fact a single JSON object:

pipeline.on("isSingleObject", () => {
  // if this event is called, the input stream contained a single object and NOT an array of objects
})

The constructur takes an options objects that can optionally contain a adjust method that is called for each object in the resulting stream before it is emitted.

anystream.addStream

A middleware for express that provides the result of anystream.make as req.anystream. The input can be one of the following:

  • A compatible data stream directly via POST data.
  • A multipart/form-data request containing a .json or .ndjson file in the field data.
  • A URL provided by the query parameter url.

An additional query parameter type, containing one of "multipart", "json", or "ndjson", may be provided if the type can't be inferred.

import express from "express"
const app = express()
import * as anystream from "json-anystream"
app.use((req, res, next) => {
  if (req.method == "POST") {
    // For POST requests, parse body with json-anystream middleware -> provides req.anystream property
    anystream.addStream(req, res, next)
  } else {
    // For all other requests, parse as JSON -> provides req.body property
    express.json()(req, res, next)
  }
})

anystream.addStream can also be called with an adjustment method first which is then forwarded into anystream.make (see above):

anystream.addStream(object => {
  // Adjust object
  return object
})(req, res, next)

Errors

json-anystream defines some custom errors:

  • MissingTypeError - thrown when the type parameter is omitted and the type could not be inferred
  • InvalidOrMissingDataFieldError - thrown for multipart/form-data when the required data field is missing or doesn't contain a .json or .ndjson file
  • UrlRequestError - thrown when the input is a URL and there's an error during the request
  • UnexpectedNonObjectValueError - thrown by the stream (i.e. for on("error")) when the stream did not contain a valid JSON object or array of objects; note that it is possible that this error is thrown after some values were already emitted in case that one of the later elements in an array is not an object

Maintainers

Publish

To publish a new version on npm after committing your changes, make sure you committed/merged all your changes to main successfully and then run:

npm run release:patch
# or for minor release:
# npm run release:minor
# or for major release:
# npm run release:major

A new version will be published to npm automatically via GitHub Actions.

Contribute

PRs accepted. Please make sure the tests complete with your change. Add new tests if applicable.

Small note: If editing the README, please conform to the standard-readme specification.

License

MIT Copyright (c) 2022 Verbundzentrale des GBV (VZG)