Gatty

I put the following in a big kettle:

• isomorphic-git
• append-only event logs
• Ink and Switch’s essay “Local-first software”
• BYOS (bring your own storage)
• browser and Node.js

then simmered the pot for a few months, and this package is what resulted. After finding that “gappy”, evoking Git and append-only, was reserved on NPM, I settled on “gatty”.

Installation

I assume you know and have Git and possibly Node.js.

Node.js setup In your Node.js project:

$ npm i --save isomorphic-gatty

and in your JavaScript/TypeScript code:

import {Gatty, setup, sync} from 'isomorphic-gatty';

Browser setup If you’re making a browser app without Node, grab index.bundle.min.js, rename it gatty.bundle.min.js and invoke it in your HTML:

<script src="gatty.bundle.min.js"></script>

It’s around 384 kilobytes unzipped, roughly 100 kilobytes gzipped.

Usage and API

Gatty is intended to support a user-facing local-first application. “Local-first” means the app keeps all user data local, and uses Gatty/isomorphic-git as one strategy for backup and multi-device data synchronization. Specifically, Gatty saves a stream of events to a git repo and synchronizes it with a remote git server (e.g., GitHub, Gitlab, Gogs, Azure Repos, etc.). The “events” are just plain strings that your app generates and understands: Gatty doesn’t know anything about them.

The envisioned use case is your app periodically calls Gatty, each time giving it the following:

• new events generated by your app (plain strings—if your app generates anything richer, JSON.stringify it first),
• a event unique identifier associated with each event—perhaps a timestamp or a random string (or both), and
• the last event unique identifier Gatty sync’d for you (empty string if you’ve never sync’d with Gatty).

Gatty in turn will return

• pairs of unique IDs & events (both plain strings) not generated on this device, i.e., by your app running on another device,
• another event unique identfier that represents the last event your app–device has synchronized, that you can use next time.

This way, the only extra thing you app keeps track of in order to use Gatty is a single stringy unique identifier.

N.B. Gatty currently doesn’t handle offline detection. Your app should make an effort to determine online status, and invoke Gatty when it has network connectivity. As we test how this works, we’ll update this section with tips.

setup

setup({corsProxy, branch, depth, since, username, password, token}: Partial<Gatty>, url: string): Promise<Gatty>

where the second argument

• url: string, the URL to clone from

is required while all the arguments of the first object are optional and passed directly to isomorphic-git:

• corsProxy: string, a CORS proxy like https://cors.isomorphic-git.org to route all requests—necessary if you intend to push to some popular Git hosts like GitHub and Gitlab, but not to others like Gogs and Azure Repos. This proxy will see your username, tokens, Git repo information, so…
• branch: string, the branch of the repo you want to work with,
• depth: number, how many commits back to fetch,
• since: Date, how far back in calendar terms to fetch,
• username: string, username for authentication (usually pushing requires this),
• password: string, plaintext password for authentication (don’t use this, figure out how to use a token),
• token: string, a token with hopefully restricted scope for authentication.

The returned value is a promisified object of type Gatty, which includes these options and a couple of other internal things.

sync

sync(gatty: Gatty, lastSharedUid: string, uids: string[], events: string[]): Promise<{newSharedUid: string, newEvents: [string, string][]}>

Given a

• gatty object returned by setup,
• lastSharedUid, a string representing the event unique identifier that Gatty told you it’s synchronized (use '', the empty string, if you’ve never synchronized),
• uids, an array of unique identifiers (plain strings),
• events, an array of events (plain strings),

Gatty will pull the latest version of the repo from the URL you gave it during setup, add the new events you just gave it, and find and returns the (promisified) events that you haven’t seen (newEvents), as an array of id–event pairs. It also returns newSharedUid, the unique identifier of the last synchronized event that you have to keep track of for future calls to sync.

Repo format

Currently Gatty will create two directories:

1. _events/, containing line-delimited JSON files:
   1. 1
   2. 2, etc. These filenames are base-36-encoded (1 through 9, then a through z, then 10, etc.). Each file contains several JSON-encoded arrays: [unique id, event text], separated by a newline. New files will be created when the last one’s size exceeds a threshold, currently 9 kilobytes.
2. _uniques/, containing one file per event. The filename is a filenamified version of the event’s unique identifier:
   1. uid1
   2. uid2, or whatever identifiers you picked. Each file contains a string: ${path to file in _events}-${number of characters to skip to get to first character of the event}. For example: _events/3-412 means: to get the event attached to this unique identifier, open _events/3 file, and skip 412 characters (JavaScript characters, i.e., UTF-16, alas).

In future, this storage format might change as we figure out which of the many deficiencies in this scheme wind up mattering most 😅.

Dev

Tape and node-git-server for local testing. (Manually tested with syncing to GitHub (see index.html).)

Browserify for bundling.

Google Closure Compiler for minification and dead-code elimination.

TypeScript for sanity.

TODO. Create a little webapp that demonstrates this with, e.g., a GitHub or something (not Gist since Gist doesn’t allow subdirectories).

Contact

If you have a GitHub account, please create an issue to get in touch, otherwise e-mail etc. is available in Ahmed Fasih’s contact info