snapshot-serializer

Big snapshot serialization/deserialization library

Problem

Imagine that you need to serialize some object (aka snapshot), which has the following form:

type Snapshot {
    metadata: {
        id: string
        author: string
        version: number
        // ... some other metadat properties
    },
    items: {
            id: number
            name: string,
            // ... some other item properties
        }[]
}

In most cases you can just do JSON.stringify(snapshot). But if you have a LOT of items you may face the following issues:

1. Total size of the result string length from JSON.stringify(snapshot) may exceed 512 MB, in that case you will face "Invalid string length" exception, caused by String length limit in JS
2. Memory usage - all the snapshot object should be loaded to the memory.
3. The total size of stringified snapshot is very big.

Solution

To solve the problem we split our initial type into the Parts. Each part can be either Array or Object type.

Array parts (in the example items) will be serialized per item and each serialized item written to file. Data for serialization provided using AsyncGenerator method, which allow to load data "by pages". Deserialization is performed by chunks - after arrayBatchSize items read, processing is started. After that, next chunk will be read and so on. That allow to avoid "Invalid string length" exception and decrease memory consumption.

Object parts (in the provided example that will be metadata) expected to have a limited size and will be serialized using regular JSON.stringify method to the file. Deserialization is performed using JSON.parse method.

Dependencies

You need to install zip and unzip utilities on your system.

Usage example

Please check tests for usage example