npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

litefs-js

v1.1.2

Published

JavaScript utilities for working with LiteFS on Fly.io

Downloads

9,626

Readme


Build Status version MIT License

The problem

Deploying your app to multiple regions along with your data is a great way to make your app really fast, but there are two issues:

  1. Read replica instances can only read from the database, they cannot write to it.
  2. There's an edge case where the user could write to the primary instance and then read from a replica instance before replication is finished.

The first problem is as simple as making sure you use a special fly-replay response so Fly can pass the request to the primary instance:

a visualization of the user making a request which is sent to a read replica and replayed to the primary instance

But the second problem is a little harder. Here's how we visualize that:

continuing the previous visualization with the edge case that the read replica responds to a get request before the replication has finished

This solution

This module comes with several utilities to help you work around these issues. Specifically, it allows you an easy way to add a special cookie to the client that identifies the client's "transaction number" which is then used by read replicas to compare to their local transaction number and force the client to wait until replication has finished if necessary (with a timeout).

Here's how we visualize that:

a visualization that shows the primary server sending a transaction number to the client and then the subsequent get request is sent to the replica which waits for replication to finish before responding

The even better (experimental) proxy solution

At the time of this writing, LiteFS just released experimental support for a proxy server that will handle much of this stuff for you. You simply configure the proxy server in your litefs.yml and then you don't need to bother with the tx number cookie or ensuring primary on non-get requests at all. The litefs-js module is still useful for one-off situations where you're making mutations in GET requests for example, or if you need to know more about the running instances of your application, but for most of the use cases, you can get away with using the proxy. Learn more about using the proxy from this PR.

Installation

This module is distributed via npm which is bundled with node and should be installed as one of your project's dependencies:

npm install --save litefs-js

Unless you plan on using lower-level utilities, you'll need to set two environment variables on your server:

  • LITEFS_DIR - the directory where the .primary file is stored. This should be what you set your fuse.dir config to in the litefs.yml config.
  • DATABASE_FILENAME - the filename of your sqlite database. This is used to determine the location of the -pos file which LiteFS uses to track the transaction number.
  • INTERNAL_PORT - the port set in the fly.toml (can be different from PORT if you're using the litefs proxy). This is useful for the getInternalInstanceDomain utility.

Usage

Integrating this with your existing server requires integration in two places:

  1. Setting the transaction number cookie on the client after mutations have finished
  2. Waiting for replication to finish before responding to requests

Low-level utilities are exposed, but higher level utilities are also available for express and remix.

Additionally, any routes that trigger database mutations will need to ensure they are running on the primary instance, which is where ensurePrimary comes in handy.

Express

import express from 'express'
import {
	getSetTxNumberMiddleware,
	getTransactionalConsistencyMiddleware,
	getEnsurePrimaryMiddleware,
} from 'litefs-js/express'

const app = express()
// this should appear before any middleware that mutates the database
app.use(getEnsurePrimaryMiddleware())

// this should appear before any middleware that retrieves something from the database
app.use(getTransactionalConsistencyMiddleware())

// ... other middleware that might mutate the database here
app.use(getSetTxNumberMiddleware())

// ... middleware that send the response here

The tricky bit here is that often your middleware that mutates the database is also responsible for sending the responses, so you may need to use a lower-level utility like setTxCookie to set the cookie after mutations.

Remix

Until we have proper middleware support in Remix, you'll have to use the express or other lower-level utilities. You cannot currently use this module with the built-in Remix server because there's no way to force the server to wait before calling your loaders. Normally, you just need to use getTransactionalConsistencyMiddleware in express, and then you can use appendTxNumberCookie as shown below.

Of course, instead of using express with getTransactionalConsistencyMiddleware, you could use await handleTransactionalConsistency(request) to the top of every loader if you like:

// app/root.tsx (and app/routes/*.tsx... and every other loader in your app)
export function loader({ request }: DataFunctionArgs) {
	await handleTransactionalConsistency(request)
	// ... your loader code here
}

The same thing applies to getEnsurePrimaryMiddleware as well. If you need or like, you can use await ensurePrimary() in every action call or any loaders that mutate the database (of which, there should be few because you should avoid mutations in loaders).

We're umm... really looking forward to Remix middleware...

The appendTxNumberCookie utility should be used in the entry.server.ts file in both the default export (normally people call this handleDocumentRequest or handleRequest) and the handleDataRequest export.

// app/entry.server.ts
import { appendTxNumberCookie } from 'litefs-js/remix'

export default async function handleRequest(
	request: Request,
	responseStatusCode: number,
	responseHeaders: Headers,
	remixContext: EntryContext,
) {
	// Most of the time, all mutations are finished by now, but just make sure
	// you're finished with all mutations before this line:
	await appendTxNumberCookie(request, responseHeaders)
	// send the response
}

export async function handleDataRequest(
	response: Response,
	{ request }: Parameters<HandleDataRequestFunction>[1],
) {
	// Most of the time, all mutations are finished by now, but just make sure
	// you're finished with all mutations before this line:
	await appendTxNumberCookie(request, response.headers)
	return response
}

Other

There are several other lower-level utilities that you can use. They allow for more customization and are documented via jsdoc. Utilities you may find helpful:

  • ensurePrimary - Use this to ensure that the server that's handling the request is the primary server. This is useful if you know you need to do a mutation for that request.
  • getInstanceInfo - get the currentInstance and primaryInstance hostnames from the filesystem.
  • waitForUpToDateTxNumber - wait for the local transaction number to match the one you give it
  • getTxNumber - read the transaction number from the filesystem.
  • getTxSetCookieHeader - get the Set-Cookie header value for the transaction number
  • checkCookieForTransactionalConsistency - the logic used to check the transaction number cookie for consistency and wait for replication if necessary.
  • getAllInstances - get all the instances of your app currently running
  • getInternalInstanceDomain - get the internal domain for the current instance so you can communicate between instances of your app (ensure you've set the INTERNAL_PORT environment variable to what appears in your fly.toml).

How it works

This module uses the special .primary directory in your Fuse filesystem to determine the primary (litefs primary docs), and the -pos file to determine the transaction number (litefs transaction number docs).

When necessary, replay requests are made by responding with a 409 status code and a fly-replay header (docs on dynamic request routing).

Inspiration

This was built to make it much easier for people to take advantage of distributed SQLite with LiteFS on Fly.io. The bulk of the logic was extracted from kentcdodds/kentcdodds.com.

LICENSE

MIT