PIFFERO

The ultimate JSON parser

Piffero is an open source SAX-like parser who work directely on the streams to get parts of big JSON files.

Piffero can load big files larger than memory used and return the required content in a stream.

  ______________________________ .  ♪
=(____|_o_o_o_o_|_o_o_o_o_|*____| |   ♫
                                 '

Description

The version 1.0.0 only support the full qualified JSONpath

example: $.father.son.array[:index].attribute

The sintax that will be implemented in the next future and the examples from Stefan Goessner's post

JSONPath | Description |Implemented -------------------|----------------------------------------------------------------------|------------ $ | The root object/element | ✅ @ | The current object/element | . | Child member operator | ✅ .. | Recursive descendant operator; JSONPath borrows this syntax from E4X | * | Wildcard matching all objects/elements regardless their names | [] | Subscript operator | ✅ [,] | Union operator for alternate names or array indices as a set | ✅ [start:end:step] | Array slice operator borrowed from ES4 / Python | ✅ ?() | Applies a filter (script) expression via static evaluation |
() | Script expression via static evaluation |

See the replit.com demo.

Get Started

Step 1: install Piffero

npm i piffero

Step 2: use Piffero

Piffero is easy to use, you only need to call one of the methods:

findByPath(inputStream: Stream, jsonpath: string): Stream

or

findAsString(callback: (result: any, err?: any) => void, inputStream: Stream, jsonpath: string): void

or

findAsPromise(stream: Readable, jsonPath: string): Promise<string>

Examples

Below there are some examples of use case based on this json file:

employees.json

{
   "employees":[
      {
         "name":"John",
         "surname":"Doe",
         "phoneNumbers":[
            {
               "type":"iPhone",
               "number":"0123-4567-8888",
               "test":false
            },
            {
               "type":"home",
               "number":"0123-4567-8910",
               "test":true
            }
         ]
      },
      {
         "name":"Joe",
         "surname":"Black",
         "phoneNumbers":[
         ]
      }
   ]
}

Example: Extract with findByPath()

For example, if you want extract the second employees "Joe Black", you need to call findByPath with the json path $.employees[1], eg:

const piffero = require('piffero');
const fs = require('fs');

// create read stream of json
const inputStream = fs.createReadStream('./employees.json');
// pass the stream to Piffero with the json path
const resultStream = piffero.Piffero.findByPath(inputStream, "$.employees[1]");

// trasform the stream to string and print into console
const resultString = '';
resultStream.on('data', chunk => resultString += chunk.toString());
resultStream.on('end', () => console.log(resultString));

console result

[{"name":"Joe","surname":"Black","phoneNumbers":[]}]

Example: Extract with findByString()

Otherwise, if you want the second phone number of the first employees, you need to call findByPath with the json path $.employees[0].phoneNumbers[1], eg:

const piffero = require('piffero');
const fs = require('fs');

// create read stream of json
const inputStream = fs.createReadStream('./employees.json');
// pass the stream to Piffero with the json path
piffero.Piffero.findAsString((result, err) => console.log(result), inputStream, '$.employees[0].phoneNumbers[1]');

console result

[{"type":"home","number":"0123-4567-8910","test":true}]

Example: Extract with findByPath() in Express.js with Typescript

import * as fs from "fs";
import * as express from "express";
import { Piffero } from "piffero";

const app: express.Application = express();

app.get("/api/:jsonPath", (req, res) => {
  
  const jsonPath: string = req.params.jsonPath;
  const inputStream: fs.ReadStream = fs.createReadStream('./employees.json');
  const resultStream = Piffero.findByPath(inputStream, jsonPath);
  
  res.setHeader("Content-Type", "application/json");
  resultStream.pipe(res);
});

app.listen(3000, () => {
  console.log("Open your browser at: http://localhost:3000/api/$.employees[0].phoneNumbers[1]");
});

PERFORMANCE

benchmarck tests was done to compare Piffero with Oboe
using benchmark

Hardware: i7-4510U 12GB RAM

jsonpath in the end of a 38 MB file

Parser | ops/secc |runs sampled -------------------|---------------------------------------------------------------|------------ Piffero (stream) | 1.0186262222075857 ops/sec | 10 Piffero (string) | 1.1235041104521386 ops/sec | 10 Oboe | 0.4287069078285109 ops/sec | 7

jsonpath in the midle of the 38 MB file

Parser | ops/secc |runs sampled -------------------|---------------------------------------------------------------|------------ Piffero (stream) | 2.192788691917704 ops/sec | 17 Piffero (string) | 2.4804025718178084 ops/sec | 16 Oboe | 0.5590235602555842 ops/sec | 7

jsonpath in the first part of the 38 MB file

Parser | ops/secc |runs sampled -------------------|---------------------------------------------------------------|------------ Piffero (stream) | 206.17873294205398 ops/sec | 68 Piffero (string) | 652.333851058612 ops/sec | 42 Oboe | 7.3452430736806225 ops/sec | 7

Piffero is almost 3 time faster than Oboe if both have to stream the whole file, the difference becomes bigger if they don't have to steam the whole file.

Other tools

• Piffero is built on Clarinet
• You can try Oboe
• You can also try JSONPath Online Evaluator

enjoy Piffero!!