Zeetl

An ETL for ZEE. Run in it Docker or import it as a Node libary.

Requirements

• Postgres
• Node 16.x or Docker

Run in Docker

The ETL can be started with Docker. Its config is set exclusively via environment variables. The following table dictates options and requirements.

| Environment Variable | Description | Default | Required | | :------------------: | :------------------------------------: | :-------: | :------: | | STAKING_PROGRAM_ID | The address of the ZEE staking program | - | true | | SOLANA_URL | URL of the Solana RPC API | - | true | | POSTGRES_PASSWORD | Postgres password | - | true | | POSTGRES_USER | Postgres user | admin | false | | POSTGRES_HOST | Postgres host | localhost | false | | POSTGRES_PORT | Postgres port | 5432 | false | | POSTGRES_DATABASE | Postgres database | zeetl | false | | SOLANA_RATE_LIMIT | Min time between API requests in ms | 250 | false | | LOG_LVL | The log level | info | false | | LOG_SILENT | Set logs to silent | false | false |

The easiest way to pass environment variables is to create a .env file containing your configuration. Here is an example:

POSTGRES_PASSWORD=dummy
SOLANA_URL=https://api.devnet.solana.com
SOLANA_RATE_LIMIT=250
STAKING_PROGRAM_ID=7vo1tfi7A7DfLi5viwb1eNwv9WUuphV2QS3TNv1nPUo5

Next, migrate the database schema.

// Insert the correct version.
docker run --rm --network host --env-file .env ormembaar/zeetl:VERSION bin/migrate

Finally, start the ETL.

// Again, insert the version
docker run -d --network host --env-file .env --name=zeetl ormembaar/zeetl:VERSION

Import as library

The ETL can also be used as a library.

const connection = new Connection('https://api.devnet.solana.com', 'confirmed');
const etl = new ETL({
    connection,
    postgresURL: 'postgres://...',
    stakeProgramId: '7vo1tfi7A7DfLi5viwb1eNwv9WUuphV2QS3TNv1nPUo5',
    rateLimit: 250
});
etl.on('error', (err) => log.error('ETL error', err));
etl.start();

Then you can query the database directly via the Prisma client embedded in the ETL.

const foo = await etl.client.stakeInitInstruction.findFirst();
console.log(foo);
/*
{
  id: 1,
  signatureId: 4,
  payer: 'FuGYQLy3jPhdUvRiAMtz71TdXwSdEVuV8xHq2XDPBAE6',
  staker: '5hWFiih4FNjGBJghozv1ZYBh4nEXyTNoCHSJgW1gg2Nj',
  stakerFund: 'BvQM3N9fw7bodh7eAjEvrmBeGkazi9wbQKHgMyEQpPGK',
  stakerBeneficiary: 'AKdjvmGqwbPQg3AtZeEzZ7fPHuDjRnXAWbznU2YzfHmt',
  communityAccount: 'ALismzdo2CZk96rAq9VMwCfRcHzhfCaA8JmjiJvc4UC',
  stakeAccount: 'BqRvpXgiT7svyn9Y83Ss6gqKkBcTECgLg7AtqnXFcFbX',
  zeeTokenMint: 'E3iGfQ6AdPLT2Q39JMNKnC78hEvoNTgqCmt7dydSAzpc'
}
*/

See the Prisma docs for help using the client. The client is generated from prisma/schema.prisma, which should be used for reference.