Testing Toolkit

🏗 A set of CLIs, tools and tests to set up, manage and operate Polygon devnets.

The Testing Toolkit is built on top of express-cli, an extension of matic-cli which uses terraform to deploy, test and monitor any devnet on AWS stacks from any local system.

It currently supports only devnets running v0.3.x stacks.

The express-cli interacts with terraform to create a fully working setup on AWS.
This setup is composed by a set of EC2 VM instances running a specific ubuntu 22.04 ami, mounted with gp3 disks , and a public-subnet with its VPC.
In case the infrastructure already exists, matic-cli can be used as a standalone tool to deploy Polygon stacks on pre-configured VMs.

Please, refer to the section of this file you are more interested in (express-cli or matic-cli)

express-cli

Requirements

To use the express-cli you have to execute the following steps.

• install aws cli
• install terraform on your local machine
• use nvm to switch to the proper node version, v16.17.1, by running nvm use from the root folder
• install express-cli and matic-cli locally with command npm i
• generate a keypair on AWS EC2 and download its certificate locally (.pem file)
• copy secret.tfvars.example to secret.tfvar with command cp secret.tfvars.example secret.tfvars and check the commented file for details
• If you are a Polygon employee, connect to the company VPN
• modify secret.tfvar with addresses of the allowed IPs (as specified in secret.tfvars.example file)
• copy .env.example to .env with command cp .env.example .env and check the heavily commented file for details
• make sure PEM_FILE_PATH points to a correct AWS key certificate, the one you downloaded in the previous steps
• define the number of nodes (TF_VAR_VALIDATOR_COUNT and TF_VAR_SENTRY_COUNT) and adjust the DEVNET_BOR_USERS accordingly
• use TF_VAR_DOCKERZIED=no to have one VM per node, otherwise the stack will run on one VM only in a dockerized environment
• (optional) replace TF_VAR_VM_NAME with your own identifier (it can be any string, default is "polygon-user")
• (optional) replace TF_VAR_DISK_SIZE_GB with your preferred disk size in GB (default is 100 GB)
• VERBOSE=true prints logs from the remote machines. If set to false, only express-cli and matic-cli logs will be shown
• If you are a Polygon employee, please refer to this page for more info

Auth Configuration

As a prerequisite, you need to configure authentication on aws
This will create the folder ~/.aws in your system To do so, please run

aws configure sso

This command will interactively ask for some configs If you are a Polygon employee, please use the following

• SSO session name: leave empty
• SSO start URL: https://polygon-technology.awsapps.com/start#/
• SSO region: us-east-1

The browser will open and authorize your request. Please allow it.

In case there are multiple accounts available to you, please select

posv1-devnet

Then, the command will ask for other configs, please use

• CLI default client Region: us-west-2
• CLI default output format: json
• CLI profile name: default

Note that it's mandatory to use CLI profile name: default, as used by terraform in express-cli (for more context see this)

Here an output example

SSO session name (Recommended):
WARNING: Configuring using legacy format (e.g. without an SSO session).
Consider re-running "configure sso" command and providing a session name.
SSO start URL [None]: https://polygon-technology.awsapps.com/start#/
SSO region [None]: us-east-1
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://device.sso.us-east-1.amazonaws.com/

Then enter the code:

<CODE-HERE>

There are 2 AWS accounts available to you.

Using the account ID <ACCOUNT_ID>
The only role available to you is: <AWSRole> (<AWS_ROLE_ID>)
Using the role name "<AWS_ROLE>"
CLI default client Region [None]: us-west-2
CLI default output format [None]: json
CLI profile name [<PROFILE_NAME_AND_ID>]: default

To use this profile, specify the profile name using --profile, as shown:

aws s3 ls --profile default

Now you can log into aws by running the following command. It needs to be executed every time the token expires.

aws sso login

Congrats! You're all set to use express-cli commands.

Commands

Instructions to run express-cli. For the list of commands, please run express-cli --help First off, you need to --init terraform on your local machine, by executing the following command.

• ./bin/express-cli --init

  • Initializes a new devnet folder with terraform and creates some git-ignored files locally. This step is mandatory before running any other command. The new devnet folder created will be devnet-<id> where id is a monotonically increasing count for the devnets. Once created, you can cd deployments/devnet-<id> and run the other commands. This allows you to work with multiple devnets at once. Then, a remote devnet can be created with the --start command, as follows.

• ../../bin/express-cli --start

  • Creates the desired remote setup, based on the preferences defined in the .env.devnet<id> file
  • --start command can be used also to target an existing AWS setup. If changes to .env.devnet<id> file are detected, the previous devnet will be destroyed and a new one created, reusing the same AWS VMs
    To destroy the remote devnet, you can execute the --destroy command.

• ../../bin/express-cli --destroy

  • Destroys the remote setup and delete the dedicated VMs

The express-cli also comes with additional utility commands, listed below. Some of them are only available for non-dockerized devnets.

• ../../bin/express-cli --update-all [index]

  • Fetches heimdall and bor branches defined as HEIMDALL_BRANCH and BOR_BRANCH in .env.devnet<id> file, pulls relative changes and restarts those services on the remote machines. If an integer index is used, the job will be performed only on the VM corresponding to that index.

• ../../bin/express-cli --update-bor [index]

  • Fetches bor branch defined as BOR_BRANCH in .env.devnet<id> file, pulls relative changes and restarts it on the remote machines. If an integer index is used, the job will be performed only on the VM corresponding to that index.

• ../../bin/express-cli --update-heimdall [index]

  • Fetches heimdall branch defined as HEIMDALL_BRANCH in .env.devnet<id> file, pulls relative changes and restarts it on the remote machines. If an integer index is used, the job will be performed only on the VM corresponding to that index.

• ../../bin/express-cli --restart-all [index]

  • Restarts bor and heimdall on all the remote machines. If an integer index is used, the job will be performed only on the VM corresponding to that index.

• ../../bin/express-cli --restart-bor [index]

  • Restarts bor on all the remote machines. If an integer index is used, the job will be performed only on the VM corresponding to that index.

• ../../bin/express-cli --restart-heimdall [index]

  • Restarts heimdall on all the remote machines. If an integer index is used, the job will be performed only on the VM corresponding to that index.

• ../../bin/express-cli --cleanup

  • Cleans up ganache, bor, heimdall and bridge, redeploys all the contracts and restarts all the services The express-cli also provides additional testing commands, listed here.

• ../../bin/express-cli --send-state-sync

  • Create a state-sync transaction on the remote network

• ../../bin/express-cli --monitor [exit]

  • Monitors the reception of state-syncs and checkpoints to make sure the whole network is in a healthy state. If --send-state-sync hasn't been used before, only checkpoints will be detected. Monitor the setup.
    If exit string is passed the process terminates when at least one stateSync and one checkpoint are detected.

• ../../bin/express-cli --instances-stop

  • Stop the AWS EC2 VM instances associated with the deployed devnet.

• ../../bin/express-cli --instances-start

  • Start the (previously stopped) AWS EC2 VM instances associated with the deployed devnet. Also, it starts all services, such as ganache, heimdall, and bor

• ../../bin/express-cli --stress [fund]

  • Runs the stress tests on remote nodes. The string fund is needed when stress tests are ran for the first time, to fund the accounts

• ../../bin/express-cli --setup-datadog

  • Sets up datadog on the nodes and gets them ready to send metrics to Datadog Dashboard. DD_API_KEY env var is required for this.

• ../../bin/express-cli --chaos [intensity]

  • Adds dedicated chaos(de-peering) to the network. The intensity parameter is optional and can be set from 1 to 10. If not set, 5 is used.

• ../../bin/express-cli --rewind [numberOfBlocks]

  • Rewinds the chain by a defined number of blocks (not greater than 128). Default numberOfBlocks value is 100.

• ../../bin/express-cli --eip-1559-test [index]

  • Executes a test to send EIP 1559 tx. In case of a non-dockerized devnet, if an integer [index] is specified, it will use that VM to send the tx. Otherwise, it will target the first VM.

matic-cli

matic-cli has to be installed on a ubuntu VM (host) and - through a config file - it will point to other VMs' IPs (remotes).

• Host machine will run a Polygon node (bor and heimdall) and a layer 1 node (ganache)
• Remote machines will only run a Polygon node each

Requirements

Please, make sure to install the following software/packages on the VMs.

• Build Essentials (host and remotes)

  sudo apt update
  sudo apt install build-essential

• Go 1.18+ (host and remotes)

  wget https://raw.githubusercontent.com/maticnetwork/node-ansible/master/go-install.sh
  bash go-install.sh --remove
  bash go-install.sh

• Rabbitmq (host and remotes)

  sudo apt install rabbitmq-server

• Docker (host and remotes, only needed in case of a docker setup)

  • https://docs.docker.com/engine/install/ubuntu/
  • https://docs.docker.com/engine/install/linux-postinstall/)

• Node v16.17.1 (only host)

  curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash
  nvm install 16.17.1

• Npm (only host)

  sudo apt update
  sudo apt install nodejs npm

• Python 2 (only host)

  sudo apt install python2 && alias python="/usr/bin/python2

• Solc v0.5.16 (only host)

  sudo snap install solc

• Ganache CLI (only host)

  sudo npm install -g ganache

Usage

On the host machine, please run

cd ~
git clone https://github.com/maticnetwork/matic-cli.git
cd matic-cli
npm i
mkdir devnet
cd devnet

Local dockerized network

Adjust the docker configs and run

../bin/matic-cli setup devnet -c ../configs/devnet/docker-setup-config.yaml

Once the setup is done, follow these steps for local docker deployment

• Move to devnet folder

  cd matic-cli/devnet

• Start ganache

  bash docker-ganache-start.sh

• Start heimdall instances (it will run all services - rabbitmq, heimdall, bridge, server)

  bash docker-heimdall-start-all.sh

• Setup bor

  bash docker-bor-setup.sh

• Start bor

  bash docker-bor-start-all.sh

• Deploy contracts on Child chain

  bash ganache-deployment-bor.sh

• Sync contract addresses to Main chain

  bash ganache-deployment-sync.sh

Logs will be stored under logs/ folder

Note: in case of docker setup, we have provided some additional scripts which might be helpful.

Remote network

Adjust the remote configs and run

../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml

Alternatively, this step can be executed interactively with

../bin/matic-cli setup devnet -i

Once the setup is done, follow these steps for remote deployment
In this case, the stack is already running, you would just need to deploy/sync some contracts, as follows:

• Move to devnet folder

  cd matic-cli/devnet

• Deploy contracts on Child chain

  bash ganache-deployment-bor.sh

• Sync contract addresses to Main chain

  bash ganache-deployment-sync.sh

Clean setup

Stop al services, remove the matic-cli/devnet folder, and you can start the process once again

Notes

1. The ganache URL hostname will be used for ganache http://<host-machine-ip>:9545
2. Make sure that the host machine has access to remote machines for transferring the data To persist ssh key for remote access, please run:

   eval "$(ssh-agent -s)"
   ssh-add `<.pem file>`

3. We have provided the default config values here to ensure smooth functioning of the process
   Please check the relative README for more accurate description of such configs These files are used as templates and dynamically modified by express-cli, hence they should not be deleted nor any modification remotely pushed Therefore, they are under .gitignore, and in case you do not want those changes to be reflected in your local git, you can use the commands

git update-index --assume-unchanged configs/devnet/remote-setup-config.yaml
git update-index --assume-unchanged configs/devnet/docker-setup-config.yaml

to undo, please use

  git update-index --no-assume-unchanged configs/devnet/remote-setup-config.yaml
  git update-index --no-assume-unchanged configs/devnet/docker-setup-config.yaml

License

MIT