Firestore Export Import

An exporter and importer for Firestore, capable of handling huge data sets, since it uses streaming and batching. Can also be used with the Firebase Emulator suite.

Installation

npm install @endran/firestore-export-import --save

or

yarn add @endran/firestore-export-import

Usage

You can use Firestore Export Import as a CLI tool, or as a library.

Use the CLI as below. --export has a higher priority then --import, which has a higher priority then --clear. Only one of the three commands will be executed.

• --export will export any document or collection denoted by ref to the file denoted by --file. Use --override to allow overwriting any file that may already exist.
• --clear erases all data denoted by ref, including all sub collection. use __ROOT__ in any command to point at the root of Firestore.
• --import writes the contents of an exported --file to firestore. Use --batchSize to limit the number of parallel promises fired at Firebase. If there is already existing data in Firestore, it will only override documents by id, and it will not erase documents in sub collections. Use --clear first if this is desirable.

fei --serviceAccount firebase-admin.json --ref "someCol" --file export.json --export --override
fei --serviceAccount firebase-admin.json --ref "__ROOT__" --clear
fei --serviceAccount firebase-admin.json --file export.json --import --batchSize 100

File format:

Firestore Export Import uses file streaming when exporting and importing. This assured that even the largest database can use Firestore Export Import. However, this also comes with a cost.

Warning: Do NOT use a formatted file for import, use the identical file syntax that is produced during export. Even though a .json file is exported, it is not treated like .json during import. The file is streamed line by line, so that it never has to be in memory completely. Formatting this file will corrupt it's structure and may lead to loss of data, or a corrupted database.

Firestore Export Import exports all documents by path. The content of each document follows the same serialization principles as node-firestore-import-export, so there is support for complex Timestamp, Geopoint and DocumentReference. However, due to version issues it will serialize any object that matches the interface, instead of a instanceof check.

Library use

If you need to export from code, use the following. To clear or import use ClearService or ImportService.

Both ExportService and ImportService also allow a transformer parameter, to modify data. This can be useful for example to encrypt personal information, per document, or to omit specific documents, by returning null. ExportService also allows for excludedCollections, this array is treated as an array of regexes, and when matched the particular collection will be skipped during export.

const exportService = new ExportService(firestore);
await exportService.export(ref, writeStream, timestamp);

// ---

export class ExportService {
    constructor(private firestore: FirebaseFirestore.Firestore) {}
    async export(
            path: string,
            writeStream: WriteStream,
            timestamp: string,
            transformer: (data: any, path: string) => Promise<any> = async data => data,
            excludedCollections: string[] = [],
        ): Promise<void>  {...}
}

export class ImportService {
    constructor(private firestore: FirebaseFirestore.Firestore) {}
    async import(
            readStream: fs.ReadStream,
            batchSize: number = Util.batchSize,
            transformer: (data: any, path: string) => Promise<any> = async data => data
        ): Promise<void> {...}
}

export class ClearService {
    constructor(private firestore: FirebaseFirestore.Firestore) {}
    async clear(path: string): Promise<void> {...}
}

Next to the content of the ref an additional metadata is added, which contains a timestamp and the ref.

Help

Usage: index [options]

Options:
  -V, --version                output the version number
  -S, --serviceAccount <path>  Required. Location of service account JSON file.
  -E, --emulator <number>      Use the FireStore emulator on the provided port.
  -R, --ref <path>             Required for --export and --clear. Reference to collection or document. Use '__ROOT__' for the root of Firestore.
  -F, --file <path>            Required for --export and --import. Location of file on disk.
  --export                     Will download ref including sub-collections to file. Has priority over import and clear.
  --import                     Will upload contents of file, merging data if required. Has priority over clear.
  --clear                      Will delete ref, and all sub-collections.
  -O, --override               Optional. If set will override any existing file during export.
  -B, --batchSize <number>     Optional. Set the amount of concurrent promises fired at Firebase. Constrains memory load for constrained devices. Use 0 to disable batching. (default: 50)
  -h, --help                   output usage information

Warning:
  Do NOT use a formatted file for import, use the identical file syntax that is produced during export.
  Even though a `.json` file is exported, it is not treated like `.json` during import.
  The file is streamed line by line, so that it never has to be in memory completely.
  Formatting this file will corrupt it's structure and may lead to loss of data, or a corrupted database.

Notes

Service Account: You can get the firebase-admin.json file from the Firebase Console. Select , click the cog, go to User and Permission, then Service Accounts, tick Node.js, and hit Generate.

Empty documents: Sadly Firestore Export Import cannot handle orphaned sub-collections. Due to performance reasons ref.listDocuments() cannot be used, instead we must use a cursor which does not return empty docs. listDocuments() can take upto 15 minutes to respond when your collection is over 100.000 documents, where as a cursor can handle each document really fast.

Keywords

Firebase, Firestore, Export, Import, Data

License

The MIT License

Copyright (c) 2019 David Hardy
Copyright (c) 2019 codecentric nl

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.