Destination Tools

Prototype tools for dev-ops to use to identify and diagnose issues with Destinations:

• Keen
• DynamoDb
• BigTable

Pre-Requisites

This application requires node.js version 11 or greater, and yarn if you want to install from the repository.

Installation

You can install from the repository and use locally.

# clone the repository
git clone [email protected]:bitbrew/destination-tools
cd desination-tools

# install dependencies
yarn

# run
./bin/dtools ls

You can also install an application from npm:

# installs 'globally' i.e. for node to run from the command line
npm i -g bbdtools

# run
dtools ls

Usage

Configuration

You can change the following parameters:

• Redis Hostname and Port
• Number of Kafka partitions in destination topics

# ordered 3 parameters, host, port and partitions
dtools config 127.0.0.1 6379 50

Example output:

Successfully changed configuration to:
Number of partitons: 50
Redis host:          127.0.0.1
Redis port:          6379

List all destinations

dtools ls

Example output:

destinationId                         closed  open  retrying  clearing  cancelled
------------------------------------  ------  ----  --------  --------  ---------
02838e3e-e1c5-434b-b91a-d03abffbbfce  50      0     0         0         0
3eeb7885-ee2a-46de-b6e1-98b202e1a486  0       50    0         0         0
86a6c111-7de6-4421-934d-81ea4fce43ee  0       1     1         49        0

Get destination status

dtools status 3eeb7885-ee2a-46de-b6e1-98b202e1a486

Example output:

Total sent records:               0
Current queued batches (queued):  125
Current queued records (rqueued): 125

Details

ptn  state  sent  retry  queued  rqueued  offset
---  -----  ----  -----  ------  -------  ------
0    Open         ✔      2       2        2
1    Open         ✔      2       2        2
2    Open         ✔      2       2        2
3    Open         ✔      2       2        2
4    Open         ✔      2       2        2
5    Open         ✔      2       2        2

Get a retry batch from a partition

dtools retry 3eeb7885-ee2a-46de-b6e1-98b202e1a486 44

Example output:

{
  "retryBatch": {
    "dipId": {
      "destinationId": "3eeb7885-ee2a-46de-b6e1-98b202e1a486",
      "partition": 44
    },
    "tenantId": "kafka-fill",
    "records": [
      {
        "eventId": "304dd66c-30e5-4e52-af8c-86c97f08f867",
        "messageId": "985c52fc-9154-4b13-a41b-d44c97e6e25a",
        "deviceId": "0000001000",
        "ruleId": "rule-id",
        "ingestionTimestamp": "2019-03-19T14:09:54.000Z",
        "payload": "{\"payload\":\"KknPvQvQV3yDklUaoztwbspMtYC3EGTzgvvkKJWJVxZc0lSWsWxSapwkTayfn0P488WX301FZEikA4OZ8FkTdiiSj7COqO9qNDi2\"}"
      }
    ],
    "retryCount": 32,
    "liveSendAt": "2019-03-19T14:12:52.000Z",
    "retryAfter": "2019-03-19T19:28:21.000Z"
  }
}

List pod assigned to partition

dtools pods

Example output

sender  ptn  pod
------  ---  --------------------------------------------------
Dynamo  0    dynamo-db-sender-v1-18-0-snapshot-5d98c55d84-dpznk
Dynamo  1    dynamo-db-sender-v1-18-0-snapshot-5d98c55d84-dpznk
Dynamo  2    dynamo-db-sender-v1-18-0-snapshot-5d98c55d84-dpznk
Dynamo  3    dynamo-db-sender-v1-18-0-snapshot-5d98c55d84-dpznk
Dynamo  4    dynamo-db-sender-v1-18-0-snapshot-5d98c55d84-dpznk
...