npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

firestore-batcher

v0.4.1

Published

Batching Firestore operations without the hassle

Downloads

813

Readme

firestore-batcher

The easier way to do batched write operations for Cloud Firestore.

npm install firestore-batcher
import * as firebase from 'firestore-admin'
import createBatcher, { deleteOperation } from 'firestore-batcher'

const app = firebase.initializeApp()
const db = firebase.firestore()

const batcher = createBatcher(db)

async function bulkDelete() {
    const finishedTodosQuery = db
        .collection('todos')
        .where('finished', '==', true)

    await batcher.all(finishedTodosQuery, (todoRef) => deleteOperation(todoRef))
}

bulkDelete()

Description

Using batches is the recommended way of performing Firestore write operations in bulk, like deleting a collection.

There are some considerations you need to make when using batch, that this library will handle for you:

  • A batch has an upper limit of 500 operations. If you need more operations, you need to make more batches.
  • Even if you stick to less than 500 ops/batch, a batch might fail due to "Transaction is too big", depending of the size of the data.

This tool will run batches recursively on your operations. The batch size is changed dynamically to handle "Transaction too big" errors.

In the following example we see that a "Transaction too big" error occurred, and the batch size was halved to 250. After that, the batch size was gradually restored to its former glory by a factor of 1.5.

Batch committed. Batch size: 500. Processed: 15013. Queued: 0. Progress: 1
Batch committed. Batch size: 500. Processed: 15513. Queued: 0. Progress: 1
Batch committed. Batch size: 250. Processed: 15763. Queued: 250. Progress: 0.9843876850059327
Batch committed. Batch size: 375. Processed: 16013. Queued: 0. Progress: 1
Batch committed. Batch size: 500. Processed: 16513. Queued: 0. Progress: 1
Batch committed. Batch size: 500. Processed: 17013. Queued: 0. Progress: 1

API

The default export is a function that takes a firestore instance, and perhaps some options, and returns a Batcher instance.

createBatcher function

(db, options?) => Batcher
import * as firebase from 'firestore-admin'
import createBatcher, { deleteOperation } from 'firestore-batcher'

const app = firebase.initializeApp()
const db = firebase.firestore()

const options = {
    onBatchCommitted: (stats) =>
        console.log(
            `Committed a batch. Total ops processed: ${stats.operationsProcessed}.`,
        ),
}

const batcher = createBatcher(db)

db (firestore.Firestore)

An instance of Firestore.

BatcherOptions object

Optional. An object of optional options for the batcher.

onBatchCommitted

(stats: BatcherStats) => void

A callback function that is called every time a batch is committed. Is called with the current stats.

Batcher

An object with various methods:

add function

<T>(operation: Operation<T>) => void

Adds an operation on a single document to the batch queue. Make sure you call batcher.commit() at least once after using this.

Example

const lotr = db.collection('books').doc('lord-of-the-rings')
const batcher.add(updateOperation(lotr, { read: true }))
const harry = db.collection('books').doc('harry-potter')
batcher.add(deleteOperation(harry))
await batcher.commit()

all function

(query: FirebaseFirestore.Query<DocumentData>, operationBuilder: <T>(doc: DocumentReference<T>) => Operation<T>) => Promise<void>

This function takes a query and a callback. The query will be limited according to batchSize, and fetched and committed recursively.

Example:

const finishedTodosQuery = db.collection('todos').where('finished', '==', true)

await batcher.all(finishedTodosQuery, (todoRef) => deleteOperation(todoRef))

If we imagine there are 1337 finished todos, the above code will first fetch 500, delete them, fetch 500 more, delete them, fetch 337 and delete them.

This method will commit each batch, so there is no need to call batcher.commit() yourself.

commit function

() => Promise<void>

Commits all queued operations recursively, using multiple batches if necessary.

stats function

() => BatcherStats

Returns the current stats.

Operations

type Operation<T> = CreateOperation<T> | DeleteOperation<T> | SetOperation<T> | UpdateOperation<T>

An operation is an object that represents a batch write operation (create, set, update or delete) on a specific document.

Four helper methods are exported to create these operation objects:

createOperation

<T>(documentRef: firestore.DocumentReference<T>, data: T) => CreateOperation<T>

Creates a new document.

deleteOperation

<T>(documentRef: firestore.DocumentReference<T>, precondition?: FirebaseFirestore.Precondition) => DeleteOperation<T>

Deletes a document.

setOperation

<T>(documentRef: firestore.DocumentReference<T>, data: T) => SetOperation<T>

Sets the data for a document.

updateOperation

<T>(documentRef: firestore.DocumentReference<T>, data: T, precondition?: FirebaseFirestore.Precondition) => UpdateOperation<T>

Updates parts of a document.

BatcherStats

{
    batchSize: number,
    operationsProcessed: number,
    operationsQueued: number,
}