Overview

This project comprises a repository interface for mutating, querying and listening to collections of strongly typed entities, with implementations for test and production scenarios.

The repository interface supports:

1. Insert
2. Update, with optional preconditions for optimistic locking
3. Delete, with optional preconditions for optimistic locking
4. Query with support for:
   1. 0..n comparison condition/s
   2. Starting offset (for pagination)
   3. Limit
   4. Order by
5. Sentinel values for server-side increment, array add/remove, timestamping and deletion
6. Transactions
7. Snapshot and delta event subscription

This interface is heavily influenced by the Google NodeJS Firestore interface. Indeed the production implementation does wrap that library, while also attempting to provide added compile-time type-safety and runtime validation. An in-memory implementation is also provided, to be used for testing scenarios.

Getting Started

The package is deployed to npm. Install as follows:

npm install @jmorecroft67/repository

Then import to your project as shown. The memory implementation is shown here or if you want to connect to Firestore see the section below for how this is done.

import { MemoryRepository, Repository } from "@jmorecroft67/repository";

interface Pet {
  name: string;
  age?: number;
  born: Date;
  children?: string[];
}

const pets = new MemoryRepository<Pet>();

async function demo(pets: Repository<Pet>) {
  const [{}, { id: fredId }] = await Promise.all([
    pets.insert({
      name: "Bingo",
      born: "serverTimestamp",
      age: 0,
    }),
    pets.insert({
      name: "Fred",
      born: new Date(2008, 1, 1),
      age: 12,
      children: ["Max"],
    }),
    pets.insert({
      name: "Lola",
      born: new Date(2009, 1, 1),
      age: 11,
    }),
  ]);

  await pets.update(fredId, {
    age: { incBy: 1 },
    children: { add: ["Mindy"] },
  });

  const snapshot = await pets
    .where("age", ">", 5)
    .orderBy("name")
    .thenBy("age")
    .get();

  snapshot.items.forEach((pet, i) => {
    console.log(`pet ${i}:`, pet);
  });
  
  // cleanup
  await Promise.all(
    (await pets.get()).items.map((snapshot) => pets.delete(snapshot.id))
  );
}

demo(pets)
  .then(() => {
    console.log("all done");
  })
  .catch((e) => {
    console.error("error:", e);
  });

At time of writing this code produced the following output:

jesss-mbp:test jessmorecroft$ npx ts-node src/index.ts 
pet 0: {
  id: '7e1c3245-7ec1-42e9-92e9-089eebcd3478',
  createdAt: 1616582027596399n,
  updatedAt: 1616582027743719n,
  data: {
    name: 'Fred',
    born: 2008-01-31T14:00:00.000Z,
    age: 13,
    children: [ 'Max', 'Mindy' ]
  }
}
pet 1: {
  id: 'ead25275-33fb-4afe-bc4d-3d11675d61d8',
  createdAt: 1616582027632841n,
  updatedAt: 1616582027632841n,
  data: { name: 'Lola', born: 2009-01-31T14:00:00.000Z, age: 11 }
}
all done

Firestore

There are a few things to be aware of when using the Firestore implementation.

Installing dependent libraries

Use of the Firestore implementation will require you install some additional libraries:

npm install @google-cloud/firestore
npm install io-ts

Setting up the (client) environment

Obviously a Firestore instance is required to connect to. You may run the gcloud emulator if you do not want to connect to the Google service. If you run the emulator you may do so directly using the gcloud CLI or another option is to run a Docker container hosting the emulator. Refer to this project's test code for the Firestore implementation for how you might run the emulator directly, or this project's GitLab pipeline config for how you might run a Docker container hosting the emulator. Regardless, if connecting to the emulator you will need to set the environment variable FIRESTORE_EMULATOR_HOST to the emulator endpoint, eg. localhost:8080.

Whether connecting to the emulator or the Google service, you should also set the project name using the GCLOUD_PROJECT environment variable.

Creating the repository instance

You must provide a schema when instantiating a Firestore repository. The schema is responsible for validating data incoming to the client from both user and backend, and also for transformation of some types to and from preferred backend types, for example JS Date and (Firestore) Timestamps. The schema is an object containing "codecs" for every required and optional member of your entity. The schema also infers the TypeScript type of your entity. The schema is defined using the io-ts library referenced earlier, so you should definitely become familiar with that if you are not already.

Here's some sample code instantiating a Firestore repository with effectively the same entity type (Pet) as the early in-memory example.

import {
  Repository,
  FirestoreRepository,
  DateFromTimestamp,
} from "@jmorecroft67/repository";
import * as fs from "@google-cloud/firestore";
import * as t from "io-ts";

const firestore = new fs.Firestore();
// Note we don't supply the Pet type here - it is automatically inferred by the schema
const pets = new FirestoreRepository(() => firestore.collection("pets"), {
  required: {
    name: t.string,
    born: DateFromTimestamp,
  },
  optional: {
    age: t.number,
    children: t.array(t.string),
  },
});

interface Pet {
  name: string;
  age?: number;
  born: Date;
  children?: string[];
}
async function demo(pets: Repository<Pet>) {
    ....
}

demo(pets)
  .then(() => {
    console.log("all done");
  })
  .catch((e) => {
    console.error("error:", e);
  });

Additional constraints on queries

Firestore imposes some additional constraints on queries compared to the in-memory implementation, which conversely allows you to query however you like - if the interface allows it of course. For example, if you run the earlier demo code with the Firestore repository, you'll find the demo fails with a rather unhelpful error.

Here it is:

jesss-mbp:test jessmorecroft$ export FIRESTORE_EMULATOR_HOST=localhost:8080
jesss-mbp:test jessmorecroft$ export GCLOUD_PROJECT=dummy-project-id
jesss-mbp:test jessmorecroft$ npx ts-node src/index.ts 
error: Error: 2 UNKNOWN: 
    at Object.callErrorFromStatus (/Users/jessmorecroft/Code/test/node_modules/@grpc/grpc-js/src/call.ts:81:24)
    at Object.onReceiveStatus (/Users/jessmorecroft/Code/test/node_modules/@grpc/grpc-js/src/client.ts:552:32)
    at Object.onReceiveStatus (/Users/jessmorecroft/Code/test/node_modules/@grpc/grpc-js/src/client-interceptors.ts:402:48)
    at Http2CallStream.outputStatus (/Users/jessmorecroft/Code/test/node_modules/@grpc/grpc-js/src/call-stream.ts:228:22)
    at Http2CallStream.maybeOutputStatus (/Users/jessmorecroft/Code/test/node_modules/@grpc/grpc-js/src/call-stream.ts:278:14)
    at Http2CallStream.endCall (/Users/jessmorecroft/Code/test/node_modules/@grpc/grpc-js/src/call-stream.ts:262:12)
    at Http2CallStream.handleTrailers (/Users/jessmorecroft/Code/test/node_modules/@grpc/grpc-js/src/call-stream.ts:392:10)
    at ClientHttp2Stream.<anonymous> (/Users/jessmorecroft/Code/test/node_modules/@grpc/grpc-js/src/call-stream.ts:436:16)
    at ClientHttp2Stream.emit (events.js:321:20)
    at ClientHttp2Stream.EventEmitter.emit (domain.js:482:12) {
  code: 2,
  details: '',
  metadata: Metadata {
    internalRepr: Map { 'content-type' => [Array] },
    options: {}
  }
}

If we look at the logs of our Docker container, where I'm running the emulator, we see what's going on.

[firestore] INFO: operation failed: found GT condition on age, but properties with inequality conditions must be the first order by element (name)

By changing our query to order by 'age' first, we fix the problem (albeit with changed behaviour) and we'll see the output as expected.

....
  // change so we order by age first, then name
  const snapshot = await pets
    .where("age", ">", 5)
    .orderBy("age")
    .thenBy("name")
    .get();
....

And now our output becomes:

jesss-mbp:test jessmorecroft$ npx ts-node src/index.ts 
pet 0: {
  id: 'msYMQk0qNjGDEALLybUO',
  data: { born: 2009-01-31T14:00:00.000Z, name: 'Lola', age: 11 },
  createdAt: 1586234020055000000n,
  updatedAt: 1586234020055000000n
}
pet 1: {
  id: 'S1w7Lg676PSwRTqJC3DX',
  data: {
    born: 2008-01-31T14:00:00.000Z,
    children: [ 'Max', 'Mindy' ],
    name: 'Fred',
    age: 13
  },
  createdAt: 1586234020061000000n,
  updatedAt: 1586234020073000000n
}
all done

Storing of dates and times

You should take care when storing datetimes in Firestore. The recommended way to do this is with the supplied DateFromTimestamp codec, which you'll see used above for the "born" schema attribute. This will work generally as you'd expect - Dates will be transformed to and from (Firestore) timestamps in the backend, which means you'll be able to deal with JS Dates in your code but when viewing the data directly in Firestore you'll see Timestamps. The big caveat however is when you use the special 'serverTimestamp' sentinel, to set or update a date to the server time value. This will still work, however as Timestamp is of nanosecond precision and JS Dates are millisecond precision, you'll likely lose precision when you retrieve the value. This is probably fine - so long as you don't overwrite it with an update - and generally why would you? - it'll still remain stored with full nanosecond precision.

On the topic of times, for any (update or delete) precondition where you set lastUpdateTime, you should never use values from the entity itself, for example some datetime attribute set using 'serverTimestamp'. Always get this value from the updatedAt attribute of the snapshot envelope for the snapshot you're updating or deleting.