Hub Shuttle Package

Package to help move data from a hub to a database.

Currently in beta. APIs should be relatively stable, but still subject to change.

However, database schema is stable and will not change (except for addition of new tables).

Architecture

There are 3 main components:

• a hub subscriber that listens to the hub for new messages
• an event processor that can process messages and write them to the database
• a reconciler that can backfill messages and ensures the messages are in sync with the hub

The event processor has a hook for your code to plug into the message processing loop. You must implement the MessageHandler interface and provide it to the system. The system will call your handleMessageMerge method for each message it processes, within the same transaction where the message is written to the db. The function is always expected to succeed, it is not possible to instruct it to "skip" a message. If your function fails, the message will not be written to the db and will be retried at a later time.

Usage

The package is meant to be used as a library. The app provided is just an example.

If you want to run the test the app, do the following:

# Ensure you have node 21 installed, use nvm to install it
nvm install 21

# If necessary, build packages/core dependency
( cd packages/core && yarn install && yarn build; )

# If necessary, build packages/hub-nodejs dependency
( cd packages/hub-nodejs && yarn install && yarn build; )

# Do remainder within the packages/shuttle directory
cd packages/shuttle
yarn install && yarn build

# Start the db dependencies
docker compose up postgres redis

# To perform reconciliation/backfill, start the worker (can run multiple processes to speed this up)
POSTGRES_URL=postgres://shuttle:[email protected]:6541 REDIS_URL=0.0.0.0:16379 HUB_HOST=<host>:<port> HUB_SSL=false yarn start worker

# Kick off the backfill process (configure with MAX_FID=100 or BACKFILL_FIDS=1,2,3)
POSTGRES_URL=postgres://shuttle:[email protected]:6541 REDIS_URL=0.0.0.0:16379 HUB_HOST=<host>:<port> HUB_SSL=false yarn start backfill

# Start the app and sync messages from the event stream
POSTGRES_URL=postgres://shuttle:[email protected]:6541 REDIS_URL=0.0.0.0:16379 HUB_HOST=<host>:<port> HUB_SSL=false yarn start start

If you are using this as a package implement the MessageHandler interface and take a look at the App class on how to call the other classes.

TODO

• [ ] Onchain events and fnames
• [ ] Detect if already backfilled and only backfill if not
• [ ] Better retries and error handling
• [ ] More tests