OWS Elastic Sync

Overview

This OWS middleware will synchronize data from a bitcoind-like system (Bitcoin, Bitcoin Cash, Litecoin, etc) to an elasticsearch cluster. This allows for distribution of queries across shards in the cluster for higher performance than can be achieved by querying a single or multiple bitcoind-like systems directly via RPC.

This middleware copies data into elasticsearch. It does not provide a query API itself. Elasticsearch can be queried directly using it's powerful API. There may be other middlewares that are developed to provide APIs for common queries.

Getting started

You can build your node using an existing OWS Node's Docker image as a base. Here is a Dockerfile to build this project for Bitcoin Cash (BCH) based on the owstack/bch-node image, built from this docker file.

FROM owstack/btc-node
USER ows
WORKDIR $APP_DIR
RUN $PKG_NAME install @owstack/ows-elastic-sync
CMD ["bchnode", "start"]

This node will also need a config file, that allows it to connect to at least 1 bitcoind-like service, and elasticsearch. Here is an example bch-node.json:

{
  "network": "livenet",
  "port": 3001,
  "services": [
    "bitcoin-cash-sync",
    "bitcoind",
    "web"
  ],
  "servicesConfig": {
      "bitcoind": {
        "connect": [{
          "zmqpubrawtx": "tcp://localhost:28339",
          "zmqpubhashblock": "tcp://localhost:28339",
          "rpcprotocol": "http",
          "rpchost": "localhost",
          "rpcport": 8339,
          "rpcuser": "someusername",
          "rpcpassword": "somepassword"
        }]
      },
      "bitcoin-cash-sync": {
          "module": "@owstack/ows-elastic-sync",
          "config": {
              "concurrentRPC": 150,
              "currency": "BCH",
              "es": {
                  "host": "localhost:9200",
                  "log": "error"
              }
          }
      }
  }
}

Lastly, an easy way to orchestrate Docker containers is with Docker Compose. A working docker-compose.yml that mounts /data from the host, might look like this (although it is not advisable to run all of this on one machine, it is possible):

version: '2'
services:
    es:
        image: docker.elastic.co/elasticsearch/elasticsearch:5.3.3
        environment:
          - cluster.name=docker-cluster
          - bootstrap.memory_lock=true
          - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
          - xpack.security.enabled=false
        ulimits:
          memlock:
            soft: -1
            hard: -1
          nofile:
            soft: 65536
            hard: 65536
        mem_limit: 16g
        cap_add:
          - IPC_LOCK
        ports:
            - 9200:9200
        volumes:
            - /data/es:/usr/share/elasticsearch/data
    bitcoin-abc:
        image: "owstack/bitcoin-abc:0.16.0-ows"
        user: root
        command: "/usr/local/bin/bitcoind -datadir=/data -printtoconsole"
        ports:
          - "8339:8332"
          - "28339:28332"
        volumes:
          - /data/bitcoin-abc:/data
    bch-node:
      image: 1234567890 #whatever your image is
      ports:
        - "3009:3001"
      depends_on:
        - "bitcoin-abc"
      volumes:
        - /data/bch-node:/home/ows/config
      command: "bchnode start -c /home/ows/config -m /home/ows/bitcoin-cash-services"

You will also need a working bitcoin.conf file in your /data/bitcoin-abc directory on the host.