npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@sudophunk/indexer

v0.0.1

Published

Indexer Monorepo

Downloads

3

Readme

This monorepo holds a set of packages and services used to deploy the V4 Indexer.

The V4 Indexer is a set of microservices which provides a set of rich APIs and websocket channels for a jInX V4 application.

The services that make up the V4 Indexer are:

  • ender: Receives on-chain events from V4 node via Kafka, archives them to a PostgreSQL database and sends websocket events for the on-chain events to Kafka
  • vulcan: Receives off-chain events from V4 node via Kafka, archives them to a PostgreSQL database and sends websocket events for the off-chain events to Kafka
  • socks: Websocket server that accepts subscriptions for websocket channels, and forwards websocket events from Kafka to websocket clients that are subscribed
  • comlink: Serves all HTTP API calls made to the V4 Indexer
  • roundtable: Periodic job service
  • bazooka: Deploys Indexer services via AWS Lambda

TODO(DEC-784): Add comprehensive descriptions/documentation for each service and the mono-repo as a whole.

Setup

We use pnpm in this repo because it is a "faster and more disk efficient package manager" and it works better than npm with a monorepo.

nvm install
nvm use
nvm alias default $(nvm version) # optional
npm i -g pnpm@6

Now, you can install dependencies for Indexer.

pnpm install

Adding Packages

Use packages/example-package as a template:

cp -r packages/example-package packages/<package-name>
  • Update package.json name to @jinxprotocol-indexer/<package-name>, update the README.md, and run pnpm i to install dependencies.
  • Add in Dockerfile.service.local, Dockerfile.service.remote, and Dockerfile.postgres-package.local to copy the package.json file and build/ files.
  • Add in Dockerfile.bazooka.remote to copy the package directory

Adding Services

Use services/example-service as a template:

cp -r services/example-service services/<service-name>
  • Update package.json name to <package-name>, update the README.md, and run pnpm i to install dependencies. Add service deployment config to docker-compose-local-deployment.yml.
  • Add Service to v4-terraform repo (TODO(DEC-838): Add link to the specific files)
  • Add Github action in .github/workflows/build-and-push.yml to push image to ECR
  • Add Service to be deployed to Orb so it will be shut down, updated, and redeployed
  • Update Auxo to deploy service

Running package.json scripts

package.json scripts for any service or package can be run with pnpm run <script-name> in the respective directory.

NOTE: pnpm allows running scripts across all directories using the -r flag or running them in parallel using the --parallel flag. Due to our unit tests sharing a test database, they cannot be run in parallel, so do not use either flag.

Protos

Protos can be found in proto/ here.

Running unit tests

First build all the services and packages by running:

pnpm run build:all

Open up 2 terminals (or have another tmux or screen session) and run:

# In session / terminal 1
docker-compose up

# In session / terminal 2
pnpm run test:all

If you change any logic, you'll have to re-build the services and packages before running unit tests.

Running Dockerfile locally

TODO(DEC-671): Add e2e tests Deploying Indexer locally serves to run e2e tests (DEC-671) and to test new messages from a local deployment from v4-protocol.

Deploying

Add to your ~/.zshrc file: export DD_API_KEY=<INSERT_DD_API_KEY_HERE>

See https://app.datadoghq.com/organization-settings/api-keys for API keys. API Key is "Key", NOT "Key Id".

Indexer can be deployed locally with: docker-compose -f docker-compose-local-deployment.yml up

If you want to export stats to Datadog, add the following flag to above command: --profile export-to-datadog

Follow steps under redeploying if any changes have been made to services since the last time indexer was deployed locally (e.g. after pulling from master/new branch).

NOTE: The kafka container may run into issues when starting up/initialization, if the kafka container is not healthy after several minutes, Ctrl-C and run docker-compose -f docker-compose-local-deployment.yml down to remove all the containers, then try running docker-compose -f docker-compose-local-deployment.yml up again.

Running local V4 node

By default the Indexer services connect to a local V4 node. To run a local V4 node alongside the indexer, follow the instructions here.

NOTE: The local V4 node needs to be started up before deploying the Indexer locally.

Connecting to remote V4 node

To change the locally deployed indexer to connect to a remote V4 node, update the environment variable TENDERMINT_WS_URL for both the comlink service in docker-compose-local-deployment.yml.

e.g.

comlink:
    build:
      dockerfile: Dockerfile.service.local
      args:
        service: comlink
    ports:
      - 3002:3002
    links:
      - postgres
    depends_on:
      postgres-package:
        condition: service_completed_successfully

Deploying to AWS Dev Environment

We use Terraform to describe our infrastructure. On merging to master branch, we automatically build ECR images of each service for both the dev/staging environments (see .github/workflows directory). However, if you want to test and deploy local changes, you can use the scripts/deploy-commit-to-dev.sh script.

Example usage: scripts/deploy-commit-to-dev.sh <commit_hash>, e.g.

scripts/deploy-commit-to-dev.sh comlink 875aecd

You can get the 7 character commit hash like so:

git add .
git commit -m "<commit_msg>"
git rev-parse --short=7 HEAD

Redeploying

The docker image for each container must be rebuilt for each code change or update to Dockerfile.local.

Images can be deleted individually:

# List all docker images
docker images
# Delete docker image
docker rmi {docker_image_id} -f

All Images can be deleted with this command:

docker rmi $(docker images 'indexer_*' -a -q) -f

The docker container can then be redeployed with the command in the previous section.

Querying API of locally run Indexer

comlink will serving API requests from http://localhost:3002.

For example, to get all BTC-USD trades:

curl http://localhost:3002/v4/trades/perpetualMarket/BTC-USD

Subscribing to websocket messages from Indexer

socks will accepting websocket connections from http://localhost:3003.

To connect to the dev/staging endpoints, use the following commands:

# dev
wscli connect ws://dev-indexer-apne1-lb-public-890774175.ap-northeast-1.elb.amazonaws.com/v4/ws

# staging
wscli connect wss://indexer.v4staging.jinx.exchange/v4/ws

Use a command-line websocket client such as interactive-websocket-cli to connect and subscribe to channels.

Example (with interactive-websocket-cli)

wscli connect http://localhost:3003
<output from ws-cli>
<type 's' to send> { "type": "subscribe", "channel": "v4_trades", "id": "BTC-USD" }

Other example subscription events:

{ "type": "subscribe", "channel": "v4_candles", "id": "BTC-USD/1MIN" }
{ "type": "subscribe", "channel": "v4_markets" }
{ "type": "subscribe", "channel": "v4_orderbook", "id": "BTC-USD" }
{ "type": "subscribe", "channel": "v4_subaccounts", "id": "address/0" }