arraybuffer-xml-parser

This code is based on a copy of fast-xml-parser.

The reason is that we wanted to parse large XML files (over 1Gb) and the current implementation of fast-xml-parser use as input a string. In the current implementation of javascript in V8 this limits the size to 512Mb.

In this code we parse directly a Uint8Array (or an ArrayBuffer) and the limit is now 4Gb.

Installation

$ npm i arraybuffer-xml-parser

Usage

XML to JSON

import { parse } from 'arraybuffer-xml-parser';

// in order to show an example we will encode the data to get the ArrayBuffer.

const encoder = new TextEncoder();
const xmlData = encoder.encode(
  `<rootNode><tag>value</tag><boolean>true</boolean><intTag>045</intTag><floatTag>65.34</floatTag></rootNode>`,
);

const object = parse(xmlData);

/*
object = {
  rootNode: {
    tag: 'value',
    boolean: true,
    intTag: 45,
    floatTag: 65.34,
  },
}
*/

By default text nodes and attribute values are dynamically typed, so 'true' becomes a boolean and '65.34' a number. Pass your own tagValueProcessor / attributeValueProcessor to keep the raw values.

Streaming a large XML

parseStream takes a web ReadableStream and yields one object per occurrence of lookupTagName, so a file larger than the available memory can be processed entry by entry.

import { open } from 'node:fs/promises';

import { parseStream } from 'arraybuffer-xml-parser';

const file = await open('medline.xml', 'r');

for await (const entry of parseStream(
  file.readableWebStream(),
  'PubmedArticle',
)) {
  console.log(entry);
}

Options

| Option | Description | Default value | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | | trimValues | Remove ascii < 32 around string values of an attribute or node. | true | | attributesNodeName | (Valid name) Group all the attributes as properties of given name. | '' | | ignoreAttributes | Ignore attributes to be parsed. | false | | ignoreNameSpace | Remove namespace string from tag and attribute names. | false | | allowBooleanAttributes | A tag can have attributes without any value. | false | | textNodeName | Name of the property containing text nodes. | '#text' | | parseAttributesString | Parse the attributes of a tag. | true | | cdataTagName | If specified, parser parse CDATA as nested tag instead of adding it's value to parent tag. | false | | arrayMode | When false, a tag with single occurrence is parsed as an object but as an array in case of multiple occurences. When true, a tag will be parsed as an array always excluding leaf nodes. When strict, all the tags will be parsed as array only. When instance of RegEx, only tags will be parsed as array that match the regex. When function a tag name is passed to the callback that can be checked. | false | | tagNameProcessor | Callback to process tag names. Receives the tag name and the matching nodes. | (name) => name | | attributeNameProcessor | Callback to process attribute names. | prefix the name with $ | | tagValueProcessor | Process tag value during transformation. Like HTML decoding, word capitalization, etc. Receives the raw Uint8Array and the current node. | decode as UTF-8 and dynamically type the string | | attributeValueProcessor | Process attribute value during transformation. Like HTML decoding, word capitalization, etc. | dynamically type the string | | stopNodes | An array of tag names which are not required to be parsed. They are kept as Uint8Array. | [] |

parseStream accepts the same options plus:

| Option | Description | Default value | | ------------- | ----------------------------------------------- | ------------- | | maxEntrySize | Maximal size (in bytes) of a single entry. | 1e7 | | maxBufferSize | Maximal size (in bytes) of the internal buffer. | 2e8 |

API Documentation

License

MIT