Firebase - Cloud Firestore Utilities

While Cloud Firestore is in Beta these utility scripts provide a CLI and an optional javascript API for exporting collections and documents as well as importing those back into your Firestore database.

Currently, these utilities do not support the following features but hopefully they will be developed in the coming releases:

• Sub-collection isolated import (export is supported when path to subcollection is given)
• Individual document import
• Multi-collection import
• Sub-collection export on individual documents
• Pause/Resume for collection import for large collections (current limit is 500 document batch writes)

Single or multiple collections can be exported to your local file system which defaults to the current working directory or to a specified file or directory path via the usage of the -o flag. Alternatively, support exists for exporting to a Google Cloud Storage Bucket. In the case of sending to a Google bucket the data is base64 encoded prior to being sent.

Overview

This CLI offers the capability to export collections from a specified Firestore database using the admin SDK.

Usage

For both import and export a path to the service account configuration can be provided as the only required argument to either command. If this value is NOT present then the application will look for the following environmental variable to generate a service config at runtime.

• GOOGLE_APPLICATION_CREDENTIALS

The above variable is required when a path to a service config is not specified. If you are running this in the context of a GCP then more can be found here.

For local development and testing of exported collections you can optionally set this variable like so:

export GOOGLE_APPLICATION_CREDENTIALS=path/to/local/service/config'`

In addition to providing a json configuration or exposing an environmental variable, you can use the -s flag which will cause the application to try and authenticate using Google Application Default Credentials to initialize Firebase. An example can be found below and additional information can be found here: Firebase Admin Setup.

admin.initializeApp({
  credential: admin.credential.applicationDefault(),
});

Command Line

Export

export [path/to/serviceAccountConfig.json]

• -c, --collections <collectionLookupPattern> - The c, collections argument can be the name of a collection at the root of the db, or alternatively a glob pattern used to perform a lookup for collections within the specified Firestore database at the root level. If neither are provided the lookup is made againt '*' so essentially all collections are exported. If a path to a subcollection is given, e.g. collection/doc/subcollection then a lookup for that subcollection is performed and, if found, only that subcollection will be exported. More information on supported glob patterns can be found in the minimatch docs If a glob is not supplied, and the CLI is used then all available collections at the root of the db are listed for selection prior to being exported.
• -d, --document <documentPath>- Specific document within a collection to export NOTE: If the document and collection are both provided then they will be concatenated and a lookup at that combined path will be used. For example, running export -c someCollection -d someDocument will result in a path lookup of someCollection/someDocument. So it is not a requirement to provide both options. Simply use export -d someCollection/someDocument.
• -o, --out <filePath>- Path to write out the contents of the exported collections. By default this is the current working directory. This path is also used with the Google Storage Bucket path if the --bucket option is supplied.
• -b, --bucket - Bucket name that the exported collections should be stored
• -g, --bucketOptions - Options for storing in Google storage. More can be found here
• -s, --defaultServiceAccount, - If set then Firebase authentication will attempt to use the default credentials which are present when running within GCP. More information can be found here
• -q, --query, Option allows for a query to be run against any returned documents from a given collection. The format of the query should be a comma separated list internally is split on that delimeter and each resulting index in the array should match the parameters described here. More on queries can be found here. NOTE Currently compound queries are not supported.

Import

import [path/to/serviceAccountConfig.json]

• -p, --filePath <filePath> - Path to collection. If this is not specified then the collection argument is required. If collection is provided and not the file path then process.cwd() is used by default and the default file extension is .jsonl.
• -c, --collection <collectionName> - name of an individual collection to be imported.

Additional information on options and usage can be found by running --help

Javascript API

Export

const firestore = require('firestoreUtils');
// initialize with the admin sdk
firestore.initialize();

// export a named collection
firestore.exportCollection('collectionId');

// export all collections based on matching glob
firestore.exportCollection('collection*');

// export a single document within a given collection
firestore.exportDocument('collectionId/documentId');

// export a single document within a given subcollection
firestore.exportDocument(
  'collectionId/documentId/subCollectionId/subDocumentId',
);

License

Firestore Utility CLI is open source software licensed as Apache License, Version 2.0.