api-integrations

Repository for managing API integrations.

Frontend

See frontend's README.

Managing deployments

• Deployment folder for the each API has the below structure:

.
├── active-deployments
└── candidate-deployments
    └── api3-20231101-airnode-feed.json

• Deployments names are created in this format: api3-YYYYMMDD-airnode-feed.json and there should be only one or no deployments under candidate-deployments.
• Once a candidate-deployment is deployed by the API provider, deployment under the candidate-deployment will be manually moved under active-deployments. active-deployments will be confirmed with CI scripts. If there is no heartbeat from the deployment, CI won't allow pushing changes to the repository.
• Deployments will identified by airnode-feed configuration file's hash

Heartbeat

Airnode feed minutely log the below string and the string will be parsed and returned through an API for this repository to check deployments.

${airnodeAddress} - ${stage} - ${airnodeFeedVersion} - ${currentTimestamp} - ${deploymentTimestamp} - ${airnodeFeedConfigHash} - ${heartbeatSignature}

Through the airnodeFeedConfigHash it will be possible to check if the correct deployment is deployed.

api-data.json

There is api-data.json file for each API provider in data/apis/<api-name>/api-data.json path. This file includes the below information.

• alias: Unique alias used for API.
• providerType: States the type of the provider. It can be either rest or rpc.
• name: Name of the API for public use.
• airnode: Airnode address of the API.
• xpub: Extended public key.
• supportedFeedsInBatches: Name of the data feeds in batches supported by API, seperated by OIS titles.
• deploymentConfigurations: Deployment configurations describes the APIs deployments.
  • signedApi is a flag that indicates if the API has an Signed API deployment.
  • region is the region that the deployment is made.
  • provider is indicates the deployment location. It could be aws or local.
• homepageUrl: API provider's website's homepage URL.
• verifiedBaseUrl: URL where includes API provider's Airnode address in it's TXT records for people to be able to be confirm the Airnode address.
• signedApiUrl: URL for the Signed API deployed by the API provider.
• productionSignedApiUrls: URLs of the Signed APIs where Airnode feeds deployed by the API provider push data to.
• stagingSignedApiUrl: URL of the Signed API where staging Airnode feed deployed by the API provider push data to.

Mock APIs

Before adding a new dAPI, we survey it for the following reasons:

• Assessing the data quality of the dAPI.
• Estimating the update count for the dAPI.

Since Airnode feed redeployments with partner API providers occur monthly, we maintain a mock version of each API that mirrors the management processes of the actual APIs. The primary difference between a mock API and an actual API is that we handle Airnode feed deployments for the mock APIs ourselves, allowing flexibility during the surveying step.

When a new dAPI is to be added to the market, the related data feeds are first added to the mock version of the actual API. After surveying, they are then added to the actual API just before the Airnode feed redeployments (Fints).

Adding a new data feed

1. Create a blueprint for the feed using pnpm add-blueprint.

> @api3/[email protected] add-blueprint /home/m3/meto_ws/api3_ws/byog/PHASE-2.1/final/api-integrations

> ts-node src/config-generation/add-new-data-feed-blueprint.ts

✔ Write data feed's name: … SOMETHING/USD
✔ Select an API to create data feed: › nodary
✔ Select the OIS: › Nodary
✔ Write the value for the parameter "path" (leave empty to pass)": … /feed/latest
✔ Write the value for the parameter "name" (leave empty to pass)": … SOMETHING/USD
✔ Write post processing snippet for "SOMETHING/USD": … (response) => { return response.value }

2. Add data feed to the corresponding supportedFeedsInBatches in corresponding api-data.json.
3. Run pnpm sync-oises && pnpm format.

Synchronizing mock and actual APIs

Mock APIs and actual APIs should be synchronized before Fints.

1. Check the differences between mock and actual APIs too see if it is as expected.

• Run pnpm ts-node scripts/check-diff-between-mock-and-actual-feeds.ts

2. Synchronize mock and actual API blueprints.

• Run pnpm ts-node scripts/sync-mock-and-actual-api-blueprints-and-api-data.ts

3. Update the OIS files.

• Run pnpm sync-oises && pnpm format

4. Do a last control.

• See the git differences.
• Re-run pnpm ts-node scripts/check-diff-between-mock-and-actual-feeds.ts

Checking providers supporting a data feed

1. Go to scripts/check-providers-for-a-feed.ts.
2. Add data feed names to feeds array.
3. Run pnpm ts-node scripts/check-providers-for-a-feed.ts.

Generating a airnode-feed deployment

1. Run the script.

pnpm generate-airnode-feed-deployment

2. Select the API.

$ ts-node src/config-generation/airnode-feed/generate-deployment.ts
? Select an API to create deployment: › - Use arrow-keys. Return to submit.
❯   api-1
    api-2
    some-api
    ...

3. Select the deployment type.

? Select the deployment type: › - Use arrow-keys. Return to submit.
❯   staging
    candidate

4. Deployment will be created under ./data/apis/<api-name>/deployments/<deployment_type>-deployments.

Integrating a new API

1. Create the below file structure under data/apis/<api-alias>.

.
├── api-data.json
├── data-feed-blueprints
├── deployments
│   ├── active-deployments
│   │   └── .gitkeep
│   ├── candidate-deployments
│   │   └── .gitkeep
│   └── staging-deployments
│       └── .gitkeep
└── oises
    └── some-new-api.json

2. Populate the api-data.json. See api-data.json reference.
3. Populate the ois file.

Initiating OIS file

1. Set oisFormat, title and version.

{
  "oisFormat": "2.3.0",
  "title": "SomeAPI",
  "version": "1.0.0",
  ...
  ...
}

apiSpecifications

2. Set apiSpecifications.components. See.

{
  ...
  "version": "1.0.0",
  "apiSpecifications": {
    "components": {
      "securitySchemes": {
        "SomeAPISecurityScheme1ApiKey": {
          "in": "query",
          "name": "apikey",
          "type": "apiKey"
        }
      }
    },
    ...
  }
  ...
}

3. Set apiSpecifications.paths. See. Regardless of the API, there should be the /{path} under apiSpecifications.paths, and it must include the parameter named path, which interpolates the /{path}.

{
  ...
  "version": "1.0.0",
  "apiSpecifications": {
    "components": {
      ...
    },
    "paths": {
      "/{path}": { <------------------
        "get": {                     |
          "parameters": [            | interpolates
            {                        |
              "in": "path", >---------
              "name": "path"
            },
            {
              "in": "query",
              "name": "<SOME_PARAMETER_OF_API_ENDPOINT>"
            },
            {
              "in": "query",
              "name": "<ANOTHER_PARAMETER_OF_API_ENDPOINT>"
            },
            ...
          ]
        }
      }
    },
    ...
  },
  ...
}

4. Set apiSpecifications.servers. See. It includes the base URL for the API.

{
  ...
  "version": "1.0.0",
  "apiSpecifications": {
    "components": {
      ...
    },
    "paths": {
      ...
    },
    "servers": [
      {
        "url": "https://api.twelvedata.com"
      }
    ],
    ...
  }
  ...
}

5. Set apiSpecifications.security. See.

{
  ...
  "version": "1.0.0",
  "apiSpecifications": {
    "components": {
      ...
    },
    "paths": {
      ...
    },
    "servers": [
      ...
    ],
    "security": {
      "SomeAPISecurityScheme1ApiKey": []
    }
  }
  ...
}

endpoints

There should be only one endpoints named feed which is designed in a way to work for multiple endpoints with operationParameter path.

6. Set endpoints[0].fixedOperationParameters, endpoints[0].name, and endpoints[0].operation.

{
  ...
  "apiSpecifications": {
    ...
  },
  "endpoints": [
    {
      "fixedOperationParameters": [],
      "name": "feed",
      "operation": {
        "method": "get",
        "path": "/{path}"
      },
      ...
    }
  ]
}

7. Set endpoints[0].parameters. This field must include the path and name parameters along with the API specific parameters. API specific parameters are referred as SOME_PARAMETER_OF_API_ENDPOINT in the following snippet.

{
  ...
  "apiSpecifications": {
    ...
  },
  "endpoints": [
    {
      "fixedOperationParameters": [],
      "name": "feed",
      "operation": {...},
      "parameters": [
        // -------- MUST INCLUDE THIS FIELD --------
        {
          "name": "name",
          "required": true
        },
        {
          "name": "path",
          "operationParameter": {
            "in": "path",
            "name": "path"
          },
          "required": true
        },
        // -----------------------------------------
        {
          "name": "<SOME_PARAMETER_OF_API_ENDPOINT>",
          "operationParameter": {
            "in": "query",
            "name": "<SOME_PARAMETER_OF_API_ENDPOINT>"
          },
          "required": true
        },
        {
          "name": "<ANOTHER_PARAMETER_OF_API_ENDPOINT>",
          "operationParameter": {
            "in": "query",
            "name": "<ANOTHER_PARAMETER_OF_API_ENDPOINT>"
          },
          "required": true
        },
        ...
      ],
      ...
    }
  ]
}

8. Set endpoints[0].reservedParameters.

{
  ...
  "apiSpecifications": {
    ...
  },
  "endpoints": [
    {
      "fixedOperationParameters": [],
      "name": "feed",
      "operation": {
        "method": "get",
        "path": "/{path}"
      },
      "endpoints": [...],
      "reservedParameters": [
        {
          "name": "_type",
          "fixed": "int256"
        },
        {
          "name": "_times",
          "fixed": "1000000000000000000"
        }
      ],
      ...
    }
  ]
}

9. Set endpoints[0].preProcessingSpecificationV2. The value field shouldn't be populated manually, it should be left blank during initializing the OIS. It will be automatically filled based on the data feed blueprints in the next steps.

{
  ...
  "apiSpecifications": {
    ...
  },
  "endpoints": [
    {
      "fixedOperationParameters": [],
      "name": "feed",
      "operation": {
        "method": "get",
        "path": "/{path}"
      },
      "endpoints": [...],
      "reservedParameters": [...],
      "preProcessingSpecificationV2": {
        "environment": "Node",
        "value": "",
        "timeoutMs": 5000
      },
      ...
    }
  ]
}

10. Set endpoints[0].postProcessingSpecificationV2. Same as the preProcessingSpecificationV2, the value field shouldn't be populated manually, it should be left blank during initializing the OIS. It will be automatically filled based on the data feed blueprints in the next steps.

{
  ...
  "apiSpecifications": {
    ...
  },
  "endpoints": [
    {
      "fixedOperationParameters": [],
      "name": "feed",
      "operation": {
        "method": "get",
        "path": "/{path}"
      },
      "endpoints": [...],
      "reservedParameters": [...],
      "preProcessingSpecificationV2": {...},
      "postProcessingSpecificationV2": {
        "environment": "Node",
        "value": "",
        "timeoutMs": 5000
      }
    }
  ]
}

11. Create blueprints for the feeds to be included in the API using the pnpm add-blueprint script.

12. Populate the api-data.json file. Ensure that the blueprints match the feeds listed in the supportedFeedsInBatches.

13. Run pnpm sync-oises && pnpm format to populate the preProcessingSpecificationV2 and postProcessingSpecificationV2 fields in the OIS file.

14. Generate a staging deployment for the integrated API.

15. Test the generated staging deployment using the local Airnode feed and Signed API deployments.

RPC providers

RPC providers are referred with providerType rpc in api-data.json of the corresponding APIs. Currently, these providers includes only Exchange Rate data feeds, it might change in the future even though there is no plan yet. Refer to RPC providers document for detailed information about how are they used in api-integrations repository.