@api3/api-integrations
v2.19.0
Published
Repository for API integration related operations.
Downloads
2,697
Readme
api-integrations
Repository for managing API integrations.
Frontend
See frontend
's README.
Managing deployments
- Deployment folder for the each API has the below structure:
.
├── active-deployments
└── candidate-deployments
└── api3-20231101-airnode-feed.json
- Deployments names are created in this format:
api3-YYYYMMDD-airnode-feed.json
and there should be only one or no deployments undercandidate-deployments
. - Once a
candidate-deployment
is deployed by the API provider, deployment under thecandidate-deployment
will be manually moved underactive-deployments
.active-deployments
will be confirmed with CI scripts. If there is no heartbeat from the deployment, CI won't allow pushing changes to the repository. - Deployments will identified by
airnode-feed
configuration file's hash
Heartbeat
Airnode feed minutely log the below string and the string will be parsed and returned through an API for this repository to check deployments.
${airnodeAddress} - ${stage} - ${airnodeFeedVersion} - ${currentTimestamp} - ${deploymentTimestamp} - ${airnodeFeedConfigHash} - ${heartbeatSignature}
Through the airnodeFeedConfigHash
it will be possible to check if the correct deployment is deployed.
api-data.json
There is api-data.json
file for each API provider in data/apis/<api-name>/api-data.json
path. This file includes the
below information.
alias
: Unique alias used for API.providerType
: States the type of the provider. It can be eitherrest
orrpc
.name
: Name of the API for public use.airnode
: Airnode address of the API.xpub
: Extended public key.supportedFeedsInBatches
: Name of the data feeds in batches supported by API, seperated by OIS titles.deploymentLocations
: Deployment locations where API provider deployed their Airnode feed to.homepageUrl
: API provider's website's homepage URL.verifiedBaseUrl
: URL where includes API provider's Airnode address in it's TXT records for people to be able to be confirm the Airnode address.signedApiUrl
: URL for the Signed API deployed by the API provider.productionSignedApiUrls
: URLs of the Signed APIs where Airnode feeds deployed by the API provider push data to.stagingSignedApiUrl
: URL of the Signed API where staging Airnode feed deployed by the API provider push data to.
Mock APIs
Before adding a new dAPI, we survey it for the following reasons:
- Assessing the data quality of the dAPI.
- Estimating the update count for the dAPI.
Since Airnode feed redeployments with partner API providers occur monthly, we maintain a mock version of each API that mirrors the management processes of the actual APIs. The primary difference between a mock API and an actual API is that we handle Airnode feed deployments for the mock APIs ourselves, allowing flexibility during the surveying step.
When a new dAPI is to be added to the market, the related data feeds are first added to the mock version of the actual API. After surveying, they are then added to the actual API just before the Airnode feed redeployments (Fints).
Adding a new data feed
- Create a blueprint for the feed using
pnpm add-blueprint
.
> @api3/[email protected] add-blueprint /home/m3/meto_ws/api3_ws/byog/PHASE-2.1/final/api-integrations
> ts-node src/config-generation/add-new-data-feed-blueprint.ts
✔ Write data feed's name: … SOMETHING/USD
✔ Select an API to create data feed: › nodary
✔ Select the OIS: › Nodary
✔ Write the value for the parameter "path" (leave empty to pass)": … /feed/latest
✔ Write the value for the parameter "name" (leave empty to pass)": … SOMETHING/USD
✔ Write post processing snippet for "SOMETHING/USD": … (response) => { return response.value }
- Add data feed to the corresponding
supportedFeedsInBatches
in correspondingapi-data.json
. - Run
pnpm sync-oises && pnpm format
.
Synchronizing mock and actual APIs
Mock APIs and actual APIs should be synchronized before Fints.
- Check the differences between mock and actual APIs too see if it is as expected.
- Run
pnpm ts-node scripts/check-diff-between-mock-and-actual-feeds.ts
- Synchronize mock and actual API blueprints.
- Run
pnpm ts-node scripts/sync-mock-and-actual-api-blueprints-and-api-data.ts
- Update the OIS files.
- Run
pnpm sync-oises && pnpm format
- Do a last control.
- See the git differences.
- Re-run
pnpm ts-node scripts/check-diff-between-mock-and-actual-feeds.ts
Checking providers supporting a data feed
- Go to
scripts/check-providers-for-a-feed.ts
. - Add data feed names to
feeds
array. - Run
pnpm ts-node scripts/check-providers-for-a-feed.ts
.
Generating a airnode-feed
deployment
- Run the script.
pnpm generate-airnode-feed-deployment
- Select the API.
$ ts-node src/config-generation/airnode-feed/generate-deployment.ts
? Select an API to create deployment: › - Use arrow-keys. Return to submit.
❯ api-1
api-2
some-api
...
- Select the deployment type.
? Select the deployment type: › - Use arrow-keys. Return to submit.
❯ staging
candidate
- Deployment will be created under
./data/apis/<api-name>/deployments/<deployment_type>-deployments
.
Integrating a new API
- Create the below file structure under
data/apis/<api-alias>
.
.
├── api-data.json
├── data-feed-blueprints
├── deployments
│ ├── active-deployments
│ │ └── .gitkeep
│ ├── candidate-deployments
│ │ └── .gitkeep
│ └── staging-deployments
│ └── .gitkeep
└── oises
└── some-new-api.json
- Populate the
api-data.json
. See api-data.json reference. - Populate the
ois
file.
Initiating OIS file
- Set
oisFormat
,title
andversion
.
{
"oisFormat": "2.3.0",
"title": "SomeAPI",
"version": "1.0.0",
...
...
}
apiSpecifications
- Set
apiSpecifications.components
. See.
{
...
"version": "1.0.0",
"apiSpecifications": {
"components": {
"securitySchemes": {
"SomeAPISecurityScheme1ApiKey": {
"in": "query",
"name": "apikey",
"type": "apiKey"
}
}
},
...
}
...
}
- Set
apiSpecifications.paths
. See. Regardless of the API, there should be the/{path}
underapiSpecifications.paths
, and it must include the parameter namedpath
, which interpolates the/{path}
.
{
...
"version": "1.0.0",
"apiSpecifications": {
"components": {
...
},
"paths": {
"/{path}": { <------------------
"get": { |
"parameters": [ | interpolates
{ |
"in": "path", >---------
"name": "path"
},
{
"in": "query",
"name": "<SOME_PARAMETER_OF_API_ENDPOINT>"
},
{
"in": "query",
"name": "<ANOTHER_PARAMETER_OF_API_ENDPOINT>"
},
...
]
}
}
},
...
},
...
}
- Set
apiSpecifications.servers
. See. It includes the base URL for the API.
{
...
"version": "1.0.0",
"apiSpecifications": {
"components": {
...
},
"paths": {
...
},
"servers": [
{
"url": "https://api.twelvedata.com"
}
],
...
}
...
}
- Set
apiSpecifications.security
. See.
{
...
"version": "1.0.0",
"apiSpecifications": {
"components": {
...
},
"paths": {
...
},
"servers": [
...
],
"security": {
"SomeAPISecurityScheme1ApiKey": []
}
}
...
}
endpoints
There should be only one endpoints named feed
which is designed in a way to work for multiple endpoints with
operationParameter
path
.
- Set
endpoints[0].fixedOperationParameters
,endpoints[0].name
, andendpoints[0].operation
.
{
...
"apiSpecifications": {
...
},
"endpoints": [
{
"fixedOperationParameters": [],
"name": "feed",
"operation": {
"method": "get",
"path": "/{path}"
},
...
}
]
}
- Set
endpoints[0].parameters
. This field must include thepath
andname
parameters along with the API specific parameters. API specific parameters are referred asSOME_PARAMETER_OF_API_ENDPOINT
in the following snippet.
{
...
"apiSpecifications": {
...
},
"endpoints": [
{
"fixedOperationParameters": [],
"name": "feed",
"operation": {...},
"parameters": [
// -------- MUST INCLUDE THIS FIELD --------
{
"name": "name",
"required": true
},
{
"name": "path",
"operationParameter": {
"in": "path",
"name": "path"
},
"required": true
},
// -----------------------------------------
{
"name": "<SOME_PARAMETER_OF_API_ENDPOINT>",
"operationParameter": {
"in": "query",
"name": "<SOME_PARAMETER_OF_API_ENDPOINT>"
},
"required": true
},
{
"name": "<ANOTHER_PARAMETER_OF_API_ENDPOINT>",
"operationParameter": {
"in": "query",
"name": "<ANOTHER_PARAMETER_OF_API_ENDPOINT>"
},
"required": true
},
...
],
...
}
]
}
- Set
endpoints[0].reservedParameters
.
{
...
"apiSpecifications": {
...
},
"endpoints": [
{
"fixedOperationParameters": [],
"name": "feed",
"operation": {
"method": "get",
"path": "/{path}"
},
"endpoints": [...],
"reservedParameters": [
{
"name": "_type",
"fixed": "int256"
},
{
"name": "_times",
"fixed": "1000000000000000000"
}
],
...
}
]
}
- Set
endpoints[0].preProcessingSpecificationV2
. Thevalue
field shouldn't be populated manually, it should be left blank during initializing the OIS. It will be automatically filled based on the data feed blueprints in the next steps.
{
...
"apiSpecifications": {
...
},
"endpoints": [
{
"fixedOperationParameters": [],
"name": "feed",
"operation": {
"method": "get",
"path": "/{path}"
},
"endpoints": [...],
"reservedParameters": [...],
"preProcessingSpecificationV2": {
"environment": "Node",
"value": "",
"timeoutMs": 5000
},
...
}
]
}
- Set
endpoints[0].postProcessingSpecificationV2
. Same as thepreProcessingSpecificationV2
, thevalue
field shouldn't be populated manually, it should be left blank during initializing the OIS. It will be automatically filled based on the data feed blueprints in the next steps.
{
...
"apiSpecifications": {
...
},
"endpoints": [
{
"fixedOperationParameters": [],
"name": "feed",
"operation": {
"method": "get",
"path": "/{path}"
},
"endpoints": [...],
"reservedParameters": [...],
"preProcessingSpecificationV2": {...},
"postProcessingSpecificationV2": {
"environment": "Node",
"value": "",
"timeoutMs": 5000
}
}
]
}
Create blueprints for the feeds to be included in the API using the pnpm add-blueprint script.
Populate the api-data.json file. Ensure that the blueprints match the feeds listed in the supportedFeedsInBatches.
Run
pnpm sync-oises && pnpm format
to populate thepreProcessingSpecificationV2
andpostProcessingSpecificationV2
fields in the OIS file.Generate a staging deployment for the integrated API.
Test the generated staging deployment using the local Airnode feed and Signed API deployments.
RPC providers
RPC providers are referred with providerType
rpc
in api-data.json
of the corresponding APIs. Currently, these
providers includes only Exchange Rate
data feeds, it might change in the future even though there is no plan yet.
Refer to RPC providers document for detailed information about how are they used in
api-integrations
repository.