web3-plugin-polkadot

A web3.js plugin for communicating with Polkadot, Kusama, and Substrate nodes.

A web3.js plugin for Polkadot, Kusama, and Substrate nodes' custom RPC methods. It provides the RPC API in the web3.js development-style. It is meant to be an alternative to @polkadot/api package.

Project name: web3-plugin-polkadot

Project creation date: 29 Nov 2023

Applying for Category 1: Polkadot ecosystem developer tools

npm package: https://www.npmjs.com/package/@conx3/web3-plugin-polkadot/

playground at: https://github.com/conx3/web3-plugin-polkadot-playground

Project website: www.web3polka.com

Introduction

As web3.js v4.x supports plugins, this library utilizes that to build a plugin for Polkadot RPC providers. This plugin allows users to execute calls like web3.polka.substrate.rpc.chain.getBlock and similar calls for all Polkadot nodes' RPCs. However, in the future, it will offer additional functionalities related to accounts & contracts, among others.

Why to build

There are 3 key benefits of this approach:

Development Experience and Learning Curve

Developers who already use web3.js would quickly start working with Polkadot and Substrate with fewer things to read and learn. All they need to do is provide the web3.js library with a link to a Polkadot, Kusama, or a Substrate provider, register the new plugin, and then start working just like they used to. There will be slight differences that they will have to adjust for. And a good progressive effort would be spent to reduce the differences to the minimum.

Code Portability

Code that already exists for Ethereum DApps, that uses web3.js, could be relatively-easily adapted to work with Polkadot and Substrate. Users would only have to make a few changes to their current code. For example, the code used to catch exceptions that indicate an unresponsive provider won't need to be changed. This also means developers may use the same code base for both Ethereum-based networks and Substrate, with just a bit of custom handling.

Diversity

It is good to provide the developers with an additional alternative to polkadot{.js}. So, developers can pick the library that they feel more comfortable working with.

Methodology

This plugin uses web3.js processing and logic; However, the types from @polkadot/types in addition to types generated by @polkadot/typegen are utilized. Using those types keeps things smooth when communicating with calling the RPC methods. In addition to keeping the available documentation for those types mostly useful, instead of inventing new types. On the other hand, having the execution inside web3.js allows for having the benefits listed in the Why to build section above.

TypeScript IntelliSense

This package utilizes IntelliSense, in order to provide the developers with the best development experience possible. So, they can use code auto-complete and hover over a method to see the method parameters and return type, and they can get a compiler error when messing up with a method name or its parameters, or when wrongly assigning its return type. Those are also already provided by polkadot{.js}. However, this plugin provides it in the web3.js style. For example, it returns the response in a Promise instead of an Observable. While Observables provide more features than Promises, Promises are simpler to learn and use, and they are aligned with the web3.js style. So, the developer can choose the library that they are more comfortable with.

Additionally, this package also limits the RPCs to what is provided for each network, especially helping the new developers get more clues about what RPC methods are available at each node.

Actually, the RPCs are accessible with the convention: web3.polka.<network>.rpc.<namespace>.<method>. Where the out-of-the-box supported network nodes are: polkadot, kusama and substrate. And, the user can add for any custom RPC and still have the IntelliSense according to the provided custom types. You may check such a code at the test file: ./test/polka-plugin-custom.test.ts

Let us take a look at how IntelliSense is provided using this plugin.

First of all, the plugin provides a list of supported networks out-of-the-box. And here are those 3:

The above list can be expanded by the developer, as you can see in the next section.

Facilitating easy access to Rpc methods, here is how the namespaces that categorize the RPC methods are listed inside the polkadot network namespace:

And, here are the RPC methods listed inside substrate network namespace:

Note that the substrate methods above are not the same as for polkadot.

And here is how a custom network that contains all the Rpcs would look like:

So, as shown above, because some RPC methods are not supported on all networks. This plugin takes this into consideration and does not show them to the developer to avoid confusion, especially for those new in this space.

However, here is how the RPC methods, inside a specific namespace, would be listed:

Picking or hovering over a specific method like getBlock would show its parameters and return type:

Hint: The types are provided by utilizing the types generated by typegen after filtering them according to the lists saved after calling the rpc_methods endpoint on the desired network nodes.

Supported Custom Rpc Methods

IntelliSense also works well with custom Rpc methods. So, if the developer registers for a specific network and names it, for example, myCustomNetwork, this is how the list of networks would appear, then:

Communicating with custom nodes or custom networks is very handy in the Polkadot ecosystem as, for example, Substrate is built to support easy configuration and customization in addition to smooth upgrades.

Here is a simple example of if a custom node only exposes Rpc methods of chain and system:

Using the plugin

To check things quickly, clone this repo, go to the test folder, update or keep the code, and then run yarn && yarn test in the terminal.

Or import and use the plugin package inside your TypeScript project following these steps:

1. In your TypeScript project run: yarn add web3 @conx3/web3-plugin-polkadot

2. After that, use something like the following code to call the RPC methods:

import { Web3 } from 'web3';
import { PolkaPlugin } from '@conx3/web3-plugin-polkadot';

// the following shows how to use the plugin
// you can call any rpc method using the convention:
// web3.polka.<network>.rpc.<namespace>.<method>
async function main() {
  const web3 = new Web3('wss://rpc.polkadot.io');
  const polkadotPlugin = new PolkaPlugin();
  web3.registerPlugin(polkadotPlugin);
  // to get the last block from Polkadot network
  const polkadotBlock = await web3.polka.polkadot.rpc.chain.getBlock();

  // call many web3 methods as usual...
  // Like updating the provider to Kusama network:
  web3.provider = 'wss://kusama-rpc.polkadot.io';
  // to get the last block from Kusama network
  const kusamaBlock = await web3.polka.kusama.rpc.chain.getBlock();

  // Updating the provider to a local substrate node for example:
  web3.provider = 'ws://127.0.0.1:9944/';
  web3.provider?.on('error', (error: any) => {
    console.error(
      'Caught provider error' +
      '(double check your running local node at port 9944): ',
      error.message || error,
    );
  });
  // to get the last block from a Substrate network
  const substrateBlock = await web3.polka.substrate.rpc.chain.getBlock();

  console.log(
    'polkadot block header stateRoot:',
    polkadotBlock.block.header.stateRoot
  );
  console.log(
    'kusama block header stateRoot:',
    kusamaBlock.block.header.stateRoot
  );
  console.log(
    'substrate block header stateRoot:',
    substrateBlock.block.header.stateRoot
  );

  // stateRoot is something like: 0xa18402bc3a2249d...

  // if you want to log the full response
  console.log(JSON.stringify(substrateBlock, null, 2));
  // {
  //   "jsonrpc": "2.0",
  //   "result": {
  //     "block": {
  //       "header": {
  //         "parentHash": "0x205d46cdcd9db4...",
  //         "number": "0x6cc7",
  //         "stateRoot": "0xa18402bc3a2249d...",
  //         "extrinsicsRoot": "0x345fc26b56a2a68...",
  //         "digest": {
  //           "logs": [
  //             "0x0661757261203b5ee81000000000",
  //             "0x056175726101011..."
  //           ]
  //         }
  //       },
  //       "extrinsics": [
  //         "0x280401000bd08620468c01"
  //       ]
  //     },
  //     "justifications": null
  //   },
  //   "id": 43
  // }
}
main();

Ensure that your node provider URLs are correct and that the respective Polkadot, Kusama, or Substrate nodes are accessible. continue provide the text according to your suggestions.

Running the tests

The tests are designed to communicate with a local Substrate node running on port 9944, and remote Polkadot and Kusama nodes are accessed at wss://rpc.polkadot.io and wss://kusama-rpc.polkadot.io. To ensure the tests pass, please make sure these requirements are met or update the URLs in the test files located in the /tests/ directory.

After cloning the repository locally and installing the dependencies, you can run the tests with: yarn test or npm test

And you should see something like:

PASS  test/web3.polka.substrate.test.ts
  test some RPC methods at web3.polka.substrate
    ✓ should call chain.getBlock method (58 ms)
    ✓ should chain.getBlock() equals 
      chain.getBlock(chain.getBlockHash(latestBlocHash)) (4 ms)
    ✓ should get the rpc methods (2 ms)
    ✓ should not have the rpc methods that is not available at substrate 
      (for example beefy.[method]) (1 ms)

 PASS  test/web3.polka.polkadot.test.ts
  test some RPC methods at web3.polka.polkadot
    ✓ should call chain.getBlock method (3885 ms)
    ✓ should chain.getBlock() equals 
      chain.getBlock(chain.getBlockHash(latestBlocHash)) (502 ms)
    ✓ should get the rpc methods (40 ms)

 PASS  test/web3.polka.kusama.test.ts
  test some RPC methods at web3.polka.kusama
    ✓ should call chain.getBlock method (487 ms)
    ✓ should chain.getBlock() equals 
      chain.getBlock(chain.getBlockHash(latestBlocHash)) (201 ms)
    ✓ should get the rpc methods (39 ms)

 PASS  test/polka-plugin.test.ts
  PolkaPlugin Tests
    ✓ should be able to register the plugin (22 ms)
    PolkaPlugin Provider exceptions
      ✓ should throw `InvalidResponseError` when the method is not supported by the provider
        (simulated) (7 ms)
      ✓ should throw `InvalidResponseError` when the method is not supported by the provider
        (calling polkadot method at substrate node) (1 ms)

 PASS  test/polka-plugin-custom.test.ts
  PolkaPlugin Custom Provider Tests
    ✓ should be able to register custom rpc methods at a custom namespace inside the plugin (38 ms)

-------------------------------------|---------|----------|---------|---------|-------------------
File                                 | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-------------------------------------|---------|----------|---------|---------|-------------------
All files                            |     100 |      100 |     100 |     100 |                   
 src                                 |     100 |      100 |     100 |     100 |                   
  index.ts                           |     100 |      100 |     100 |     100 |                   
  polka-plugin.ts                    |     100 |      100 |     100 |     100 |                   
 src/types/constants                 |     100 |      100 |     100 |     100 |                   
  kusama-supported-rpc-methods.ts    |     100 |      100 |     100 |     100 |                   
  polkadot-supported-rpc-methods.ts  |     100 |      100 |     100 |     100 |                   
  substrate-supported-rpc-methods.ts |     100 |      100 |     100 |     100 |                   
-------------------------------------|---------|----------|---------|---------|-------------------
Test Suites: 5 passed, 5 total
Tests:       14 passed, 14 total
...

What has been accomplished and what is next

Implemented features

During the limited time, those are the implemented features:

• A full functional web3.js plugin that provide access to all RPC methods in the web3.js style.
• Limit the RPC methods to what is available at polkadot, kusama and substrate, out-of-the-box.
• Developers can provide a their list and the types specifications of the supported RPC methods for any variant network or customized node and has the IntelliSense accordingly.
• Tests with 100% coverage.
• The npm package has been published to the npm registry.
• A basic documentation has been provided in this readme file.

Planned Features

• Implement additional functionalities, including subscription, account handling, and smart contract interactions. Actually, moving further would require sometimes inheriting from web3.js classes to modify their behaviors accordingly. But, the modified version should continue in providing, as close as possible, api to web3.js. On the other hand, when possible, a PR would be opened at web3.js to suggest enabling the desired customization if it make sense allow their.
• Develop more plugins for Polkadot that could be used with this plugin. For example, a plugin that allows the developer to pass the smart contract code at runtime and immediately start interacting with it would be feasible, similar to this plugin: https://www.npmjs.com/package/web3-plugin-craftsman. Such implementations would give the developers a variety of options. So, the web3.js plugins can be utilized together to provide the optimal development experience.
• Apply for funding:
  • At the treasury spending proposals: https://polkadot.network/ecosystem/treasury/
  • Or getting a Web 3 Foundation Grant: https://grants.web3.foundation/docs/introduction for the Decentralized Futures.

Do you like to Contribute?

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to add or change.

Please make sure to update tests as appropriate when contributing.

Notes:

• Types have been generated by following the process detailed here: https://polkadot.js.org/docs/api/examples/promise/typegen/
• Local Substrate node setup instructions can be found here: https://docs.substrate.io/quick-start/start-a-node/

Team member information

Muhammad Altabba (solo) - Blockchain Tools Developer

License

MIT